Scaling Up and Moving In : Connecting social practices views to policies and programs in adult education

The social practices framework has had a major impact on adult literacy and numeracy research over the past quarter century in the US, the UK and other countries. To date, the social practices view has had far less influence on the development of policies and programs in adult literacy and numeracy education. To help this happen, new kinds of assessment tools aligned with the social practices framework are needed to support appropriate changes in curriculum design, learner assessment and program evaluation. In this article research is presented that illustrates how measures of adults’ engagement in literacy and numeracy practices can be used in conjunction with well-entrenched proficiency measures to provide a richer quantitative framework for adult literacy and numeracy development. Longitudinal data about learners indicate that adult education programs are more closely aligned with practice engagement measures than with proficiency measures. Program participation leads to increased practice engagement that, over time, leads to the very gains in proficiency currently valued by policy makers.


Introduction
The social practices framework for literacy has had a major impact on adult literacy and numeracy research over the past quarter century.An accumulating body of research in this framework has created a new discourse about literacy, one that can effectively stand in opposition to the 'dominant' institutionally-based discourse about adult literacy.Impressive as this research is, the social practices view has had far less influence on the development of new policies and programmatic practices in adult literacy and numeracy education (Hamilton 2001).Although the new understandings and discourses about adult literacy and numeracy generated by the social practices approach offer an important foundation for change, there is great need to develop corresponding innovations in curriculum design, learner assessment and program evaluation that will help adult education programs and practitioners deal with pressing practical needs.
Although proponents of the social practices approach have developed strong and persuasive critiques of the interpretive and policy schemes that rely on standardised literacy test scores (e.g.Street 1997, Hamilton and Barton 2000, Hamilton 2001), alternatives have not been proposed that are practical for use on a large scale.There are many reasons this has not yet happened.The development of the social practices framework -as with the development of critical theory more generally -has relied on rich qualitative analyses of behaviours, discourses and texts in highly localised settings.Some theorists are understandably hesitant to embark on a methodological journey that they fear might produce only narrow and reductionist measures of social practices.That is clearly a danger and something that must be guarded against.
A second impediment may be the view that because literacy practices are locally situated, they can be meaningfully measured and interpreted only within local contexts.Although this may be true for some literacy and numeracy practices, many other practices are constructed over broader contexts.Indeed, some literacy practices have been sociohistorically constructed specifically to transcend the limits of the local (Brandt andClinton 2002, Reder andDavila 2005).
Another barrier may be the widespread polarisation of qualitative and quantitative methods in many fields of educational research, including research on adult literacy and numeracy.Erickan and Roth (2006) argue against an extreme polarisation and propose an integrated approach that creates a continuum rather than a dichotomy of methods and generalisability.They suggest that the types of research questions being asked should determine the modes of inquiry (drawn from this continuum) used to answer them.Luke and Hogan (2006) provide a model for school-based research in Singapore that draws on a range of qualitative and quantitative methods that cohere within a social practices framework of teaching and learning in schools.
Luke and Hogan's work is far more than just another call for the use of mixed methods in complex research projects.They assert it is possible to adopt what they term a 'critical realist' approach to evidence and theorybuilding that can embody the core principles of critical theory while interfacing with the complex embeddings and nested organisational structures of educational institutions.I argue that, for research on adult education, an analogous approach needs to be developed, one that acknowledges and takes into account the diverse settings, contexts and identities associated with adult literacy and numeracy practices and programs designed to foster them.
There is some reason to believe that such an enterprise could connect practically with programmatic concerns.Previous research has suggested that measures of literacy practices are related to adults' experiences in basic skills programs while measures of literacy proficiency are not.Sheehan-Holt and Smith (2000) examined the large-scale survey data from the National Adult Literacy Survey conducted in the US in 1992, contrasting the results for adults who participate in basic skills programs with results for those who do not.With education and numerous other background variables statistically controlled, their multivariate analyses found evidence of program impact on literacy practice measures but not on literacy proficiency measures.
The use of practices-based measures cannot only provide evidence of program impact, it can identify effective programmatic models for adults.Purcell-Gates, Jacobson and Degener (2004) found that students from adult education programs that focus instruction around authentic literacy materials and practices report greater improvements in their literacy practices than students from programs not centered around authentic literacy practices.Although each of these studies suggests that systematic use of practices-based measures may be invaluable for understanding and improving programs, neither utilised measures satisfactory for use in longitudinal work, essential for addressing these types of research questions (Reder and Bynner 2008).
In this article, I will take a few initial exploratory steps in this direction.I will describe some key previous findings and present new analyses from a project that has followed adults over long periods of time, observing changes in their lives, in their literacy and numeracy practices, and in their standardised proficiency scores.By contrasting analyses based on measures of practices and proficiencies, I hope to show that carefully constructed practices measures can offer a stronger and more practical platform for the development of adult education policy and programs than a platform based on proficiency measures alone.
The remainder of this article is organised into four sections.The first section will introduce the Longitudinal Study of Adult Learning (LSAL).The second section will consider a few key recent findings from LSAL, identifying important differences between proficiency and practices measures.A major question emerging from these findings is whether increased engagement in literacy and numeracy practices over time leads to increased proficiency.A positive answer to this question is crucial for making the case that measures of adult literacy and numeracy practices should be systematically used for developing better policies and programs in adult education.The third section conducts new analyses of the LSAL data to address this question.The final section discusses the findings emerging from these analyses and their implications for reframing the dominant discourse about adult literacy and numeracy from its narrow focus on proficiency to a broader focus that includes literacy and numeracy practices.

The Longitudinal Study of Adult Learning
The Longitudinal Study of Adult Learning (Reder 2007, 2008, Reder and Strawn 2001a, 2001b, 2006) examines the nature and impact of literacy and numeracy development across the lifespan of youth and adults who dropped out of high school in the US.This multimethod project followed a panel of

Population and sample
The LSAL followed a target population for adult education defined as residents of the Portland, Oregon metropolitan area, aged 18-44, proficient but not necessarily native English speakers, high school dropouts (i.e., did not receive a high school diploma and were no longer enrolled in school) and without a General Educational Development (GED) or other high school equivalency credential.ii A statistically representative sample of this population was drawn from two sampling frames: random-digit-dialing for the general population and enrolment forms from the three major adult education programs serving the Portland metropolitan area.Sampled households were called and screened for members in the defined target population.The resulting sample contained 940 individuals.Complex sampling weights were used to construct population estimates from sample data.
At the beginning of the study in 1998, the population had an average age of 28 and was evenly divided among males and females.Approximately one-third were members of minority groups, one in ten were born outside of the US, one third described themselves as having a learning disability, and one in three reported having taken special education classes (designed for students with physical and learning disabilities) while they were in elementary or secondary school.Individuals dropped out of school for a variety of reasons.

Interviews and assessments
Six periodic interviews and skills assessments were conducted in respondents' homes.Respondents were paid for each of these sessions, which took an average of about one and a half hours to complete.Each wave of data collection consisted of an in-home interview followed by cognitive assessments.The skills assessments included a standardised functional literacy assessment in each wave.This proficiency measure was the Document Literacy scale of the Test of Applied Literacy Skills (TALS).Administered in a constructed response rather than multiple choice format, the TALS assesses adults' abilities to extract and process written information in a variety of everyday document formats, such as forms, maps, tables, text displays, labels, and so forth.These written documents are utilised to perform simulated everyday literacy tasks.Respondents are assigned proficiency scores on a 0-500 scale based on the simulated literacy tasks they are able to perform correctly.The TALS instruments are highly similar to instruments used in the 1992 National Adult Literacy Survey, the 2003 National Assessment of Adult Literacy, the International Adult Literacy Survey, the Adult Lifelong Learning survey and in numerous state surveys in the US.
Self-reported measures of everyday reading, writing and math activities were collected each time respondents were interviewed.They were asked about how often they performed each of a set of specific reading, writing, numeracy and computer practices in various everyday contexts (home, community, work).Two questions were asked about each practice.Respondents were first asked if they ever engaged in the practice, (e.g.'Do you ever read the news section of the newspaper'?).If respondents answered 'yes', they were asked a second question about their frequency of engaging in that practice (e.g.'How often do you read the news section of the newspaper?'),answering on a five point scale ranging from 1 (rarely) to 5 (every day).Answers to the pair of questions for each practice were combined so that the possible range of scores for each practice was from 0 (never) to 5 (every day).
Confirmatory factor analysis was carried out on these practices data.iii Fourteen practices from the home and community contexts were included in this analysis.Confirmatory factor analysis is a highly sensitive technique that provides information about the degree to which items measuring the same concept are closely related as well as information about whether there are multiple concepts measured by the set of items.Using this analytic approach, we eliminated items that tended to be poorly associated with the other items, and we discovered that two central concepts were assessed by the remaining items, which we termed engagement in literacy practices and numeracy practices.
After finalising this confirmatory factor model with the Wave 1 data, we conducted a series of analyses to ensure that the measurement properties of the scales were stable across waves of the study.It is essential that measurement properties are stable across waves for valid longitudinal analyses of change.Without stable measurement properties, it is impossible to distinguish changes in literacy practices from changes in the measurement properties of the scale.iv In order to establish longitudinally stable measurement properties, we substantially reduced the number of items used per scale, so that the resulting scales are measured with considerable error (in terms of classical psychometric criteria) on any one occasion.For repeated measures analyses, however, such measurement error may be acceptable if it is stable over time.
The final literacy practices scale was comprised of three practices (how often fiction was read; how often non-fiction was read; how often notes, letters or emails were written), and the numeracy practices scale was comprised of two items (how often maths for bank accounts, credit cards, etc. was used; how often maths was used at home).It is noteworthy that the particular items comprising these longitudinally stable scales were relatively broadly specified practices (e.g.'How often do you read fiction'?)rather than more narrowly specified practices.

Some Key Recent Findings from LSAL
This section briefly summarises a few key findings from LSAL about adults' development of literacy and numeracy proficiency and practices.These findings are based on statistical modeling of the repeated measures of proficiency and practices collected over Waves 1 through 5 (Reder 2008).

Changes in proficiency
Assessed proficiencies vary widely in the LSAL population but are relatively stable within individuals over time.About 75% of the variance in scores in this large data set is located between individuals as opposed to within individuals over time.Nevertheless, there are statistically significant and systematic changes in individuals' proficiencies over time, and more importantly, significant heterogeneity among individuals' rates of proficiency change: Some individuals gain proficiency over time, others lose proficiency, whereas others maintain their proficiency at a roughly constant level.Individual rates of proficiency growth are generally higher among younger adults and immigrants than among older and native-born adults.
Understanding the heterogeneity of change in proficiency -why some individuals gain proficiency in adult life while others lose proficiency -is likely to be very important for adult literacy policy and program design.Key life events and economic forces seem to influence proficiency growth over the lifespan.For example, changes in an individual's employment and earnings are associated with observed changes in their proficiencies; individuals gaining employment or increased earnings tend to show increasing proficiencies, and vice-versa (Reder forthcoming).
Although changes in proficiency appear closely related to changes in economic activity, no relationship was observed between proficiency change and participation in adult basic skills programs.This is a striking finding because current accountability regimes hold programs accountable for producing short-term 'learning gains' on proficiency measures when, according to these results, programs do not have short-term effects on this type of literacy measure.v

Changes in practices
Statistical models, parallel to those constructed for the repeated measures of proficiency, were developed for the literacy and numeracy practices measures.Comparing the models reveals some important similarities and differences between proficiency and practices measures.To begin with, the measures are positively correlated -individuals with relatively high levels of proficiency also have relatively high levels of engagement in literacy and numeracy practices.All of the measures exhibit systematic change over time as well as significant heterogeneity of change among individuals.Although there is substantial correlation among individuals' levels of proficiency and practices, there is not significant correlation between their rates of change over time in those levels.The proficiency and practices measures are subject to quite different dynamics of change (Reder 2008).
One important difference among the measures is how they reflect program impact.Although there is not a clear relationship between program participation and changes in proficiency, there is a strong relationship between program participation and changes in practices.With many statistical controls in place, Reder (2008) found strong relationships between participation in adult education programs and changes of engagement in literacy and numeracy practices.The temporal sequencing of the observed changes makes it clear that program participation influences practices rather than vice-versa.This finding is consistent with the relationship that Purcell- Gates et al (2004) reported between types of adult education programs and changes in practices observed among program participants.Purcell-Gates et al found that adult education students from programs that focus instruction around authentic literacy materials and practices report greater changes in their literacy practices than students from programs not centered around authentic literacy practices.LSAL contrasts the development of literacy and numeracy practices between program participants and non-participants whereas Purcell-Gates et al contrast the development of literacy practices among participants in different types of programs.
These findings pose a critical dilemma for adult education programs.On one hand, programs have demonstrable impact on measures of literacy and numeracy practices but not on proficiency measures, at least not in the short-term.At the same time, proficiency measures are at the very core of the dominant discourse that justifies investments in programs in terms of the increased proficiency and associated economic benefits they produce.Can we reconcile these two findings and argue that programs should be designed and evaluated in terms of the increased engagement they produce in literacy and numeracy practices?Elsewhere I have suggested the possibility that programs in the short-term produce increased engagement in practices, and that those higher levels of engagement in literacy and numeracy practices might lead over time to increased proficiency (Reder 2008).Although this may seem a reasonable possibility, there has not been direct evidence to date to support it.We next turn to a new analysis of the LSAL data that examines this possibility directly.

Practice Engagement Theory and the LSAL data
Practice engagement theory (Reder 1994, Sheehan-Holt andSmith 2000) holds that engagement in literacy practices leads to growth in literacy proficiency.Since LSAL collected parallel information about proficiencies and practices over time, it is possible to test the predictions of practice engagement theory with the LSAL data.One way to do this is with the classic simplex modeling approach developed by Humphreys (1960), Joreskog (1979) and others.We look at relationships among the three variables of interest -proficiency, literacy practices and numeracy practices -at two different points in time, Wave 1 (1998) and Wave 5 ( 2004).We model how the relationships among the three variables change between the two time points.Each of the three key variables is allowed to influence the other two, so that reciprocal influences or effects are allowed among the variables between Waves 1 and 5.These can be lagged effects, that is, the effect of a variable at Wave 1 on its own Wave 5 counterpart, as well as cross-lagged effects between a variable at Wave 1 and another variable at Wave 5. Wave 1 values are assumed to be predetermined within this simplex framework, with correlations allowed among the three measures.These potential lagged effects, cross-lagged effects and correlations are shown in Figure 1 on page 43.
We are particularly interested in the cross-lagged effects.Given that the lagged effects of each variable are already taken into account in the model, a significant cross-lagged effect would reflect the influence of one variable on another over time (e.g the effect of earlier levels of engagement in literacy practices on later levels of literacy proficiency, a practice engagement effect).The simplex model shown in Figure 1 is estimated by structural equation modeling.Statistical tests are used to evaluate the estimated path coefficients and the overall goodness of fit of the model to the data and as well as to compare the fit of alternative models.I estimated four alternative specifications of the general simplex model shown in Figure 1.Results are summarised in Table 1.Model A is the fully saturated simplex model, meaning that there is a set of coefficient values that will fit the observed covariance data perfectly.Thus, the goodness of fit for Model A is not of interest (it will be perfect), but the fitted values of the lagged and cross-lagged coefficients that produce the perfect fit will be of considerable interest.All estimates shown in the table are standardised, so that coefficients correspond to effect sizes.In Model A, each of the adult literacy and numeracy variables has a significant lagged effect on its subsequent value, reflecting the relative stability of these measures of proficiency and engagement in practices over long periods of adult life.The proficiency measure has a considerably larger lagged coefficient than does either of the practice engagement measures, reflecting its higher level of stability (Reder 2008).Only one of the cross-lagged effects is statistically significant, the effect of Wave 1 engagement in literacy practices on Wave 5 proficiency.This significant (t=2.355,p=.019) cross-lagged effect is theoretically important, being consistent with practice engagement theory.More frequent reading and writing activities lead over a long period of time to greater proficiency.Interestingly, we do not see significant cross-lagged effects between numeracy practices and proficiency nor between numeracy and literacy practices.We do note a marginally significant (t=1.734,p=.083) cross-lagged effect of the earlier level of proficiency on the subsequent level of engagement in numeracy practices.Weak to moderate correlations are also observed between the three measures at Wave 1, consistent with Reder's (2008) earlier findings.
Model B removes the non-significant cross-lagged effects from Model A. Because Model B is not fully saturated, its goodness of fit to the observed data can be evaluated.Two customary goodness-of-fit measures from structural equation modeling are shown for Model B in Table 1.The small value (0.017) of the root mean square error of approximation (RMSEA) and the large value (0.990) of the Tucker-Lewis Index (TLI) indicate that Model B fits the observed data quite well.vi The coefficients estimated for the two cross-lagged effects in Model B are quite similar to their counterparts in Model A. The literacy practices-to-proficiency effect is still significant (t=2.851,p=0.004), whereas the proficiency-to-numeracy practices effect remains only marginally significant (t=1.807,p=0.071) To investigate further the marginally significant effect of proficiency at Wave 1 on engagement in numeracy practices at Wave 5, we compare the overall fit of Models C and B. Model C drops the marginal cross-lagged proficiency-to-numeracy practices effect from Model B. The fit of Model C is still quite good judging from its RMSEA and TLI indexes.Since Model C is nested within Model B, the difference in the models' chi-square values is distributed as chi-square with a single degree of freedom.vii The scalecorrected chi-square difference has a test statistic of 1.278 with one degree of freedom, which is not statistically significant.Model C thus fits the data as well as Model B does, and since it is more parsimonious, we prefer Model C.
Model D adds seven time-invariant covariates to Model C: • age (at Wave 1) • gender • years of education (before dropping out) • minority status The effects of these covariates on each of the three literacy measures are estimated at Wave 1 and at Wave 5.The basic structure of Model C is not affected by adding these covariates to Model D. The practice engagement effect -leading from engagement in literacy practices to literacy proficiencyremains significant with numerous demographic and background variables controlled.Two significant Wave 5 covariate predictors are worth noting here.Age has a significant negative effect on Wave 5 proficiency with Wave 1 proficiency controlled.This is consistent with previous findings of lower rates of proficiency growth among older individuals (Reder 2008).Parental education has a significant positive effect on Wave 5 engagement in literacy practices with Wave 1 engagement levels and other variables controlled.Family background, such as parents' education, influences the growth of engagement in literacy practices across the adult life course.

Discussion and conclusions
Because of the problematic ways that key issues in adult literacy and numeracy have been framed by the dominant discourse, it is important for the social practices perspective to increase its practical leverage on adult education policies and programs.It seems unlikely that large-scale publiclyfunded programs will operate in other than a quantitative framework.Although proficiency measures will likely retain a place in any such framework because of their close empirical relationship to schooling and economic status, additional measures are needed that better reflect a social practices perspective and that are more closely aligned with the impact that programs actually have on adult literacy and numeracy development.Adding appropriate practices-based measures to policy and programmatic frameworks would broaden and enrich policy-makers' perspectives on adult literacy and numeracy development and lead to more effective programs.This article has taken some initial steps to facilitate movement in this direction.First, differences were highlighted between proficiency and practices measures as indicators of adult literacy and numeracy development.Previous research shows that proficiency and practices measures have distinct dynamics of change that differentially reflect the influences of program participation, life events and economic forces.A key concern is that policies and public investments are frequently rationalised in terms of programs' presumed or measured impact on adults' proficiency, while research shows programs have their most direct and immediate effect on adults' literacy and numeracy practices.There is thus a major misalignment between the effects programs are having on their students' literacy and numeracy development, on one hand, and the short-term proficiency gains for which programs are accountable under the dominant policy and funding regimes.As the stakes rise in these accountability schemes, such misalignments are likely to produce substantial distortions in educational practice.
A second step demonstrated the relevance of practice engagement measures to policy and programmatic concerns.Analyses of recent longitudinal data provide clear evidence of practice engagement effects on long-term proficiency development.Adults at similar proficiency levels at one point in time wind up many years later at different proficiency levels depending in part on their earlier levels of engagement in literacy practices.Those with higher levels of engagement at an initial point in time have higher levels of proficiency at a later point in time even with initial levels of proficiency controlled.There is also strong evidence that programsespecially programs utilising authentic materials and practices in the classroom -foster higher levels of engagement in literacy practices in their students that persist after they leave the programs.There is thus a strong chain of evidence linking programs to increased engagement in practices and linking practice engagement over longer periods of time to increased proficiency levels.Without the use of literacy practices measures, a systematic connection is not evident in these data between programs and proficiency.We can make the case for adopting such measures with the demonstration that increased levels of practice engagement -something that programs produce -have consequences valued by policy makers, that is, increased proficiency.
As I suggest that adult literacy and numeracy education systematically adopt the use of practices-based measures, let me hasten to emphasise that great care must be taken in developing and using these measures.The particular practice-engagement measures presented in this article are not necessarily recommended.They were not developed for such administrative purposes, they were developed for longitudinal research.Our experiences developing these measures, however, may provide some useful lessons for developing other such measures.The temptation to develop narrow and reductionist practices measures must be resisted -we do not want to create an 'autonomous' model of adult literacy and numeracy practices.The measures adopted should reflect learning outcomes that are broadly important to adults as well as to policy-makers and funding agencies.They should be carefully developed from a theoretically driven social practices perspective.The measures should be based on literacy and numeracy practices that occur in a broad range of social contexts and geographical settings rather than on highly localised practices.
Development and use of such measures, of course, faces many methodological and interpretive challenges.The social practices perspective has systematically evolved through ethnographic inquiries focused on the meaning rather than the frequency of actions, an approach that does not readily fit with the concept of a psychometrically sound measurement scale.In this regard, it is interesting that our experience building longitudinally stable measures required the use of contextually broad rather than contextually narrow and discrete practices.We must also be vigilant in remembering that 'more' engagement in social practices is ultimately worthwhile as a policy objective only if it is 'better' for the individuals involved.Keeping these limitations in mind, we should carefully move ahead with the enterprise of adding practices-based measures to the policy and programmatic frameworks in adult education.Although the social practices perspective gives us good reason to proceed cautiously, the 'critical realist' perspective urges us forward.