Program Evaluation as Community-Engaged Research Challenges and solutions

Much of the thinking that has shaped our understanding of community-engaged research has its roots in the intellectual résistance of the early 20th century to the domination and permeation of the positivist imagination into our ways of knowing, talking about and interacting with the world. Since then, positivism’s vision of a singular source and mode of knowledge, accessed, secured and enriched only through carefully guarded instruments and methodologies, has been systematically assaulted by social theorists at two major points. At the first of these, adherents of the critical theory tradition have attacked the tendency of positivist models to reduce diversity and complexity in the social world to rationalised taxonomies and empirically observed social ‘facts’, querying the possibility of their ‘objective’ observation and even disputing their very existence. Elsewhere, cultural and political theorists have exposed the myriad ways in which positivist epistemologies preserve and rely upon systems of bureaucratic control, sustaining the privileges of the elite and entrenching social stratification and inequality. These critiques, intellectual progenitors of post-modernist epistemologies, instead seek to promote plural and local forms of knowledge and envisage sociological research as a function of emancipation. Their authors, from Michel Foucault to Edward Said, have inspired a re-imagination of research as a collaborative, inclusive and impact-driven process that acknowledges the role of knowledge creation in the establishment, preservation or disestablishment of the latent power relationships that sustain disadvantage and social division. In the hands of critical pedagogists such as Paulo Freire and Henry Giroux, this epistemological tradition became a tool for those committed to the empowerment of the disempowered, the granting of voice to the unheard, and the inclusion of the excluded. Theories of community-engaged research emerged from within this tradition as an approach to research conducted in community contexts, and encouraged the development of collaborative strategies for advancing community wellbeing, in so doing seeking to foster Gateways: International Journal of Community Research and Engagement Vol 8/No 1 (2015): 118–38 © UTSePress and the author

and support partnerships between 'researchers' and 'researched' characterised by two-way learning built on a commitment to knowledge exchange and mutual respect and recognition.
Despite being a tool for understanding impact and effectiveness, evaluative research into community-based programs has not always taken this approach.Indeed, for those whose programs or activities are 'under evaluation' it might give the impression of reinforcing control rather than advancing collaboration, equity, mutual learning or any other emancipatory outcome.In part, this is an inevitable consequence of differences and disagreements vis-a-vis evaluation terminologies, methodologies and strategies.On the face of it, evaluation has a number of broad goals, serving principally to guide program development, support institutional planning and enhance accountability.In practice, the weighting or priority these goals receive depends hugely on context.
In considering the relationship between evaluation and the principles of community-engaged research, we also need to think briefly about an important semantic question.The extant body of literature on evaluation practice implies some degree of consensus that evaluation, in non-academic contexts, employs unique techniques that set it apart from other forms of social research -and thus that there may be a clear difference between evaluation and evaluative research.To an extent this is undeniably true -the methods evaluation deploys are often focused first and foremost on delivering findings that are useable and have practical applications; this often requires 'compromising' on traditional concerns for research quality vis-a-vis data validity in order to address certain professional and practical expediencies.
Notwithstanding the emergence of quite different languages and approaches, however, to some degree this is also something of a false dichotomy, one that has emerged from and speaks to differences in professional positioning and structures rather than fundamentally different methods, values or goals.
While recognising this is still a live debate, for the purposes of this present article I assume a degree of interchangeability between the terms evaluation and evaluative research.I therefore urge the reader to see beyond any language that suggests an alignment with one mode over the other to the conceptual and practical issues that are important considerations for all modes and forms of evaluation.From this premise I advance the contention that evaluative research into social interventions can -and indeed should -be both conceptualised and operationalised as engaged research committed to effecting positive social change.The article frames its reflections in the context of evaluative research into a number of outreach programs at Macquarie University, Sydney, targeted at school students and community members from backgrounds that are underrepresented within Australia's higher education student population.The article illustrates ways in which evaluative research can be conceived of in terms of community engagement by mapping research strategies against two influential models of community engagement.It then reflects on some of the challenges in implementing these ideal practices within the context of evaluative research, recognising that the challenges and opportunities that have arisen during this evaluative research reinforce the conclusion that such models can only provide aspirational targets.In this sense, the appropriate moral and professional framework for such research is one that combines a commitment to engagement with a reflexive, adaptable, pragmatic and above all iterative approach to methodology and stakeholder relations.

BACKGROUND: EVALUATING OUTREACH ACTIVITIES
The evaluative research upon which the reflections in this article are based focused on seven educational interventions led by Macquarie University professional and academic staff.
These programs are part of a broad portfolio of activities at Macquarie funded by the Federal Government's Higher Education Participation and Partnerships Program, a tranche of bespoke funding initially conceived as a response to a review of Australia's higher education system in 2008 that identified the disproportionately low participation of a number of societal groups in higher education, including individuals from low socioeconomic, Aboriginal, and culturally and linguistically diverse backgrounds.
Collectively, the Macquarie programs have identified a number of critical objectives to assist the goal of ensuring that the representation of these groups in higher education corresponds to their demographic representation within the broader population, including raising ambitions and aspirations, increasing capacity and skills, and tackling structural disadvantage.
In order to assess progress against these aims, in 2012 Macquarie commissioned evaluative research to complement existing reporting and monitoring exercises and develop strategic understanding of the programs' impacts.This evaluation consisted of both formative and summative components and produced five principal outcomes: 1 In its formative aspect, the evaluation provided a 'reflective space' in which evaluators were able to work with program facilitators to help identify appropriate, evidence-based improvements.The evaluation sought to explain not just what the impact had been, but also how that impact was achieved, allowing program facilitators to build on the successes and strengths of individual programs and to address critical weaknesses.
2 The research findings supported strategic decision-making around the University's social inclusion and widening participation strategies.
3 By contributing to various reporting processes, the evaluation supported efforts to ensure the university meets its legal accountability and transparency requirements.
4 The evaluation helped to construct an evidence base on the impact of social inclusion activities, thus building a case for the continuation of funding and public support and interest.
5 The evaluation contributed to general knowledge concerning disadvantaged students from a range of backgrounds and the barriers they face in accessing higher education.
The collection and analysis of data was guided by a theoretical and methodological framework that fused current best practice in evaluative research with the specific aims and objectives of the program.The evaluation was holistic in scope, intended to gather triangulated data but also to uncover evidence of how impact is often mediated by the relationship between and contributions of the various stakeholders in the program.

EVALUATION AS COMMUNITY-ENGAGED RESEARCH
Although the increasing demand for accountability and transparency of publicly funded social interventions has inspired a focus on summative evaluation (Shah, Nair & Wilson 2011), the relatively developmental character of many of the programs under assessment at Macquarie (as well as the portfolio to which they belonged) meant it was important to give equal attention to formative aspects of the process.Formative evaluation accrues a number of benefits above and beyond those of summative evaluation; in the present case it was anticipated that it would facilitate learning and program development (Harris, Jones & Coutts 2010;Nesman, Batsche & Hernandez 2007), as well as build evaluative and reflexive capacity -critical given the limited duration of the evaluative research (Preskill & Boyle 2008;Smeal, Southwell & Locke 2011).Formative evaluation, of course, is also particularly conducive to participatory methods and models of engaged research (Harris, Jones & Coutts 2010;Hashimoto, Pillay & Hudson 2010;Stoner et al. 2012).However, the research also advanced from a recognition of the potential for top-down disengaged research to extend and deepen entrenched disadvantage, and thus a failure to engage would risk working against the goals of the programs it was seeking to assess.
There were a number of additional reasons why the principles of engaged research were seen as crucial to this evaluation project.The exchange of knowledge facilitated by collaborative partnerships is critical to ensuring that evaluative practice is directly informed by the reality of conditions on the ground.Procuring the wisdom of multiple perspectives and the intimate knowledge of program development is key to the development of effective, efficient, ethically sound and minimally disruptive evaluation instruments.It should also not be overlooked that the trust engendered by the good relationships created as part of engaged research practice ultimately enhances the possibility of more intimate and accurate testimony from research participants who understand and buy into the function and purposes of the research.However, we would do well to recognise that, while partnerships characterised by high levels of trust are likely to provide more intimate testimony, better understanding of the rationale and direction of the research project may also lead to a bias in results as research participants provide answers in interviews and focus groups they feel the researcher wants to hear.
In creating the conditions and building the relationships that enable and sustain knowledge exchange, engaged research also facilitates the building of capacity (evaluative capacity, in this case).External evaluation is rarely cheap, even where it is readily available.Moreover, whatever expertise or putative objectivity external evaluation brings to the table, it takes time for the evaluators to familiarise themselves with evaluated projects, and requires effective communication on an ongoing basis to ensure researchers are aware of developments in the program under evaluation.The inevitable distance of the external researcher from the program also means that there is a constant risk that evaluation findings are misaligned with program objectives or otherwise misrepresent the work under evaluation.
Internal evaluation, by contrast, drastically reduces the risk of this disconnect because it is informed by professional experience in the program.For these reasons a number of authors have argued that best evaluative practice by necessity should include a capacity-building component.Again, engaged research with a focus on the building of collaborative relationships characterised by trust, mutual learning and knowledge exchange are plainly the best vehicles for this process (Nesman, Batsche & Hernandez 2007;Oliver et al. 2002;Preskil & Boyle 2008;Ryan, Chandler & Samuels 2007;Smeal, Southwell & Locke 2011).
From the outset, therefore, the evaluative research at Macquarie envisaged engaged methodologies as critical to its success, thus embedding a commitment to collaboration in the key research stages of design, implementation, iteration, analysis and dissemination.The research identified individual program staff, principals and senior teaching and support staff in schools, and parents and community representatives as key stakeholders and, as far as possible, encouraged and presented opportunities for these stakeholders to contribute to research processes.The inclusion of stakeholder voices at every stage of the research was also intended to ensure that the findings of the evaluation were a product of the authentic integration of multiple areas of expertise and local knowledge, brought together within and through the research process.This input would help to maximise the research's potential to empower its participants to make informed decisions about program improvements and professional or pedagogic practices, and to equip its participants with a collaboratively constructed evidence base to support program sustainability and advocacy efforts.
With these considerations in mind, due thought was given from the outset to how theories of community engagement might support the development of engaged and inclusive research strategies and methodologies for the Macquarie evaluation.
This was not a simple proposition: the current literature on community engagement is emergent rather than established, and the frameworks that do exist are varied in quality, detail, scope and applicability.& Huffman 2003).This was followed by an initial design phase that involved further consultations around matters of methodology and anticipated implementation; during this stage, and subsequent iterative design stages, stakeholders were asked to comment on proposed interview schedules and to contribute questions of their own that would provide useful information in their professional development.
The stakeholders were also critical in the implementation stage.Not only did they broker contacts between the researchers and the program participants, but in line with the ambition to raise evaluative capacity they were also directly involved in the data collection (though decisions in this regard also had to be balanced with matters of research ethics and data integrity).

MITIGATING IMPACT
Notwithstanding these aspirations, there were a number of challenges that inhibited the evaluation's success in realising this idealised framework on the ground.Figure 3 provides a visual representation of some of the key issues: it imagines a series of challenges and opportunities that collectively constitute 'intervening factors' that mediated the impact of the evaluation as an engaged project.
The challenges the evaluative research faced varied tremendously, from those that were relatively easily overcome to those that required significant compromises.Many of these problems were anticipated as part of an initial risk identification and management process, but this itself became an evolving strategy as the initial approaches adopted to minimise these risks proved unsuccessful or insufficient.These challenges may be usefully considered within the framework of five broad categories: definition, paradigm, participation, resources and governance.
Challenges of definition concerned issues of boundaries and inclusion.One of the core challenges of engaged research into the sorts of large multi-agency projects included in this evaluation was the difficulty in identifying the full range of stakeholders (that is, the targets for engagement).The programs under evaluation all involved a number of groups and individuals who could be legitimately considered stakeholders based on their capacity to affect or be affected by the programs (e.g.see Bryson 2004, p. 22), foremost among whom were program funders and coordinators, in addition to a range of teachers, school administrative and support staff, community representatives, parents, school and university student participants, and volunteers.In some cases, however, identifying the stakeholders was less straightforward.
The National Indigenous Science Education Program (NISEP), for instance, involves training school students to present scientific

MEETING THE CHALLENGES
Notwithstanding the tendency for these challenges to sometimes seem like insurmountable hurdles, the extant community engagement and evaluation literatures offer a number of strategies for overcoming such difficulties.For instance, one tangible response to the challenges of definition is a robust stakeholder analysis process that identifies the interest and power of key stakeholders, their relationships with one another and the program, and their Similarly, a good stakeholder analysis will include an assessment of stakeholders' strategic concerns and interests, and thus provide a strong platform for the development of specific methods for increasing the engagement of sceptical or disinterested stakeholders.
The Macquarie evaluation made good use of some of these strategies, and in many cases the experience of the research also reinforced their value.Many of these strategies formed part of the systematic consultations the researchers initiated with the program teams during the research planning phase, guided by the RUFDATA evaluation planning tool developed at the University of Lancaster (Saunders 2000).In accordance with this model, the researchers engaged the program coordinators and key stakeholders in a series of structured conversations designed to establish a number of key baselines for the evaluation: its key objectives (reasons and uses), what activities it will evaluate (foci), the evidence it intends to collect (data), key stakeholders and dissemination plans (audience), appropriate timescales (timing) and who is responsible for which aspects (agency).
The conversations were drafted into a series of collectively owned written agreements.This process provided information that allowed the researchers to make informed decisions about inclusion strategies, as well as how to overcome the problem of high-influence stakeholders with more complex orientations towards the programs and the evaluations.It also constituted a form of stakeholder analysis that provided the rationale by which engagement with stakeholders who were most directly impacted by, or had the greater influence over, the program could be prioritised over those with less influence or interest.
There are additional strategies researchers might use to boost stakeholder engagement.For instance, researchers might fuse stakeholder analysis with a strengths-based framework to counter the tendency for stakeholders to feel unqualified, thus identifying approaches to the problem of researcher passivity (Harvey 2014).
Establishing channels for ongoing consultation throughout the course of the research -not just in the initial planning phases -is also critical.The establishment of a community advisory group with strong, inclusive leadership based on a solid understanding of power dynamics, for instance, can create a space in which stakeholders can air concerns and respond to emergent challenges; and it can also boost the stakeholders' sense of agency and ownership of the research project.The establishment of subcommittees and inclusive approval or feasibility processes may also help to increase levels of active engagement by increasing the individuals' sense of agency in the process.
Where issues of resources make such structures difficult to construct and maintain, the researchers might instead commit to planned iteration phases that prompt renewed consultations and reflections with key stakeholders.This was the preferred option in the case of the Macquarie research, where the rather complex nature of the research, its various programs, and the wide range and diversity of stakeholders made it extremely difficult and time consuming to establish formal advisory structures.
Instead, the formalised periods of 'iterative program design' provided prompts and opportunities for re-engagement and also presented critical opportunities for realigning the research where changes in program delivery or personnel over time had left a distance between the program and its evaluation.The scheduled consultations that were part of this iterative process helped to ensure the research stayed relevant to its original goals and abreast of the influence of these changes.
Reflection on the evaluative research at Macquarie also exposed additional complexities that suggest we should implement such processes with a degree of care and sophistication.
Sometimes, for instance, there remain nuances such processes (or the literature they are based upon) do not always address.In conducting stakeholder analysis, for example, researchers need to find ways to distinguish between the identities and characteristics of stakeholders in the program and stakeholders in the research.
Ultimately, ensuring research methods are implementable and conducive to supporting community-engaged research principles (such as capacity strengthening and knowledge exchange) throughout the lifetime of the research means understanding the nuances of the stakeholders' orientations towards the research itself, not just the program.There is often a good degree of overlap between the two, but at other times there are some important differences that any stakeholder analysis should address.
The Macquarie evaluation also proved that the task of conducting a stakeholder analysis can be complicated by a whole series of specific contextual factors.For instance, it became apparent that high levels of support among practitioners for the In many cases, the researcher has a key role to play here: overcoming paradigmatic differences, for instance, requires that researchers 'demonstrate inclusion and respect to the fullest extent possible' (Gittelsohn et al. 2003) by respecting community customs and practices.Even where the research has clearly constrained goals and methods, ensuring the community are included may well require a genuine ethnographic engagement, where researchers attend events and functions, for example, not directly related to the research but that nevertheless serve an important role in building trust and respect.There are also various practical solutions the researcher might implement to help minimise potential conflicts and misunderstandings where stakeholders are situated in different professional paradigms.Face-to-face meetings, for instance, are inevitably better than conference calls or emails in this regard, opening up a broader range of communicative cues and extending the possibility of finding shared ground.
Misunderstandings about research practice might be overcome via a commitment to knowledge exchange realised through training or capacity-strengthening activities on the one hand and the researcher's participation in community events on the other.
Ultimately, however, effective relationship building requires reciprocity; no engaged project can succeed without all sides respecting and recognising the experience and expertise each side brings to the table.Overcoming paradigmatic differences requires that all parties maintain an awareness of the potential for miscommunication and be scrupulous about the assumptions they make.Often the literature puts much of the onus on the researcher to take responsibility for crafting such relationships and to make the critical compromises necessary for ensuring these relationships develop.In community contexts, this is often based on the perception that the researcher occupies a privileged power position.In reality, however, even when engaged with significantly disadvantaged communities, the power relationships are complex -community gateholders, for instance, can wield significant influence over the outcome of the research.Rather than taking full responsibility for the success or failure of these relationships, the researcher might be better served to think of their role as creating the conditions for such relationships to emerge (such as open communication channels).A researcher who makes too many compromises may find him or herself becoming less rigorous in observing their inclusive ideals as frustration mounts; in striving to manage the responses, fears or interests of others, the researcher should not forget to attend to his or her own.
In practice, it may be more effective to craft a middle ground where certain ground rules are respected.Often this can be as simple as observing meeting etiquette, a small but deceptively important aspect of conveying mutual respect and recognition.
Creating professional structures and interactions characterised by mutual respect and recognition of each stakeholder's experience and expertise is a critical foundation, and a commitment to regular communication will help build awareness of the limitations and pressures on both sides (Horn et al. 2008).
Ultimately, the most effective relationships will always emerge where both sides are committed to this process and have a shared This array of challenges also makes one thing particularly clear: good intentions are not enough.The reflections herein disclose the need for researchers to deploy a range of tools and to take on a number of roles, serving as brokers and mediators and being prepared to spend at least as much time building relationships and exchanging knowledge as on more traditional research activities such as data collection and analysis.Here lies perhaps one of the greatest challenges for engaged evaluative research: it places a huge onus on the researchers, not only in terms of time commitments, but also in terms of the array of skills they must bring to the table.In addition to research-specific skills, the evaluator must have knowledge of evaluation methods and content-specific knowledge, and be able to act as a broker, mediator and educator, adapt his or her language to a diverse range of audiences, and constantly translate, mediate and bridge professional discourses.The requirement for such a broad range of skills is why some writers stress the value of using evaluation teams (Worthern & Sanders 2011).
Of course, not every challenge is surmountable.Some may need to be accepted or worked around rather than overcome.
Many of the challenges researchers face in implementing engaged evaluative practices relate to the difficulty synthesising divergent forms of knowledge, language, professional practice and agendas.
The researcher's success in uniting these often determines the level of impact evaluative research has, but the lack of clear consensus as to the appropriate response to paradigmatic conflict makes this a tricky and indeterminate business.There is also perhaps more outside the researcher's control than we might ascertain from consulting best practice literature.The reality is that the quality More importantly, however, implicit in its commitment to build confidence, capacity and trust is the potential to open up future, as yet unimagined, possibilities.
Figure 1: National Indigenous Science Education Program evaluation mapped against Arnstein's Ladder (1969)

Finally, stakeholders were
schemes and a mentorship program in partnership with a national

Figure
Figure 2: Evaluative research against the five 'goals' of community engagement

Figure 3 :
Figure 3: The mitigation of impact: challenges and opportunities programs under assessment could translate into a more opposed orientation towards evaluative research -or indeed any research perceived as intrusive or threatening.Most of the staff involved in the research programs at Macquarie were employed for fixed terms, with no guarantee of employment beyond the end of their term, a reality that is true of many social interventions and makes for a particularly fraught and complex context in which to conduct evaluative research.As a consequence, it was not always straightforward to establish stakeholders' level of interest and influence in the research, or their true orientation, particularly where there was thought to be a connection between being seen to offer outward support for the research and ongoing employment or other professional opportunities.The author certainly experienced occasions during the evaluations where key stakeholders expressed strong outward support for the evaluative process, but failed to follow through on promises of support -or even worked against the research to a degree behind the scenes.Confronting as they may be, such challenges are an inevitable aspect of social and professional environments characterised by intersecting and competing interests, hopes and fears of different stakeholders, requiring some triangulating of sources and a degree of resilience and creativity on the part of the researcher.These added complexities remind us that, while thorough planning and inclusive structures and processes are important tools for enabling stakeholder engagement, there is no solely procedural panacea to the typical challenges of communityengaged research.Reflecting on the limitations with some of these processes as they manifested through this evaluative research helped to reaffirm the critical foundation provided by effective relationship building, enabled through strong communication and interpersonal skills.Solid professional relationships built on mutual trust, respect and recognition, and characterised by transparency and authenticity, can go a long way to overcoming key participation and paradigmatic differences.They can also open up avenues for the building of evaluative capacity, empowering stakeholders to deliver positive impact themselves on later occasions that might not have been possible without the research project.And, by facilitating mutual learning and knowledge exchange, effective professional relationships can do much to help to mobilise additional resources and smooth the processes of decision-making associated with effective research governance.
appreciation for the value such relationships add.The effectiveness of this process is inevitably increased where practitioners and managers provide structures and incentives to support it.At the very least, it should be recognised and acknowledged in the planning stages, prior to the research being commissioned, that relationship building will need extra time and, where possible, work collaboratively with the researchers to build support for and understanding of the research process.SIDEWAYS THINKINGThere are occasions, as the Macquarie research evidenced, when even these responses are insufficient to overcome the challenges of community-engaged research and deliver both tangible research outcomes and the sort of full and inclusive stakeholder engagement to which this model of research aspires.When solutions have been exhausted, or where relationships just do not function no matter how much energy has been invested from either or both sides, community-engaged research can feel like an uncomfortable, uncertain and unmanageable enterprise.This author certainly encountered such moments, and contended regularly with uncertainty as to how to overcome differences in understanding and attitudes associated with individual personalities or long professional or cultural legacies -factors that proved ultimately beyond the power of the researchers to address satisfactorily within the project's confines.These occasions remind us of an important reality of community-engaged research.'No battle plan', insisted the Prussian military strategist Helmut von Moltke, 'survives first contact with the enemy'; so too it is impossible to plan away all the potential complications and challenges that arise when conducting community-engaged research.Planning, strategising and doing all one can to build effective collaborative structures and spaces remains an essential part of conducting engaged research, but often the most appropriate response to these challenges is for the researcher to address his or her own attitudes and expectations (and, where relevant, those of the commissioning organisation).Sometimes this involves adjusting (or perhaps resisting, or at least navigating around) some of the traditional ways in which research quality is recognised within the academy.For instance, where questions of data validity are wedded to positivist conventions, there may well be a corresponding desire to strictly control independent variables.But to embark on a communityengaged research project means to an extent learning to become comfortable with a certain amount of fluidity, uncertainty and compromised objectives.Quite apart from being antithetical to the principles of community-engaged research, strict control is rarely possible in any collaborative context.During the Macquarie research, two particular approaches proved particularly useful in countering the temptation to look for 'perfect' solutions and outcomes.The first was to give preference to post-positivist qualitative methods that described alternative modes of knowledge creation.Where quantitative methods sometimes struggled to keep abreast of the changing variables that contributed to the evidence of impact, taking a qualitative, narrative-building approach allowed the evaluative research to meaningfully identify and discuss areas of impact without seeing cause and effect as purely a matter of measuring variables against evidence of change.The second response was to explicitly anchor the research in the values that underwrote both the program and the community-engaged research model, so that the research aimed for a broad vision of impact determined not just by the quality of its data but also by the quality of its relationships and the useability of the findings.This wider approach to the question of research 'goals', 'impact' or 'success' provided a strong foundation that also enabled the researchers greater flexibility in responding to changes in program practices or personnel.This last approach was aided by a little lateral thinking.For instance, notwithstanding the depth and breadth of the challenges mapped above, and their potential to seriously impact upon the successful implementation of engaged evaluative research models, during the Macquarie evaluation a number of unanticipated opportunities for the research to maximise its impact emerged when the challenges were reconsidered from another angle.For instance, where evaluators came across negative or misinformed perceptions of research practice that hindered participation, there was a corresponding opportunity, if that research was conducted in a democratic and inclusive spirit, to go some way towards repairing those perceptions.By combating negative pre-perceptions, engaged research might be seen as a means of facilitating future research activity, even if the immediate results have been limited.Perhaps not what those who commission research are always ready to hear, but certainly a tangible outcome with very worthwhile long-term benefits.One final unanticipated opportunity evidenced by the evaluation of the Macquarie programs has been the potential for evaluative practices to directly contribute to program objectives.As discussed above, one of the key aims of the Macquarie outreach programs was to build capacity and confidence within student participants.There is indicative narrative and anecdotal evidence from the Macquarie research that focus groups and interviews requiring participants to reflect on their participation in the program and comment on possible improvements have both helped to develop critical, reflexive thinking and to make the students feel included and heard.A second key objective was the building of capacity within professional staff (in the schools, in this case); the evidence of the formative evaluation processes has been used for professional development purposes and to guide and inform pedagogic practice.Moreover, as attested to by the Macquarie-based facilitators, much of the programs' success depends on the strength of the relationships between community and school-based stakeholders and the Macquarie staff.Through consultations with the stakeholders, the evaluation process has thus itself helped to deepen and strengthen those connections upon which the programs depend.CONCLUSIONThe reflections in this article offer a small contribution to our understanding of the complex relationships and processes that characterise the intersection of academic research, educational interventions, and community and school-based pedagogic practices.In a pragmatic and conceptual sense, these frequently manifest as critical challenges for the researcher committed to synthesising the values that inform these different practices via models of community-engaged research.These challenges keep the engaged researcher honest: like any community-engaged research, managing a community-engaged evaluative project involves striving to secure a balance between maximising opportunities and remaining ever mindful and attentive to the attendant risks.
of engagement in evaluative research, like any form of research, may often benefit from the intervention of third parties, such as the line managers of stakeholders, to help motivate or encourage stakeholder participation.Ultimately, this discussion reinforces the argument that there can be no 'perfectly' realised model of community-engaged research.Processes and techniques for overcoming key challenges all tend to manifest a common acceptance of the need to prioritise resources and energies; once we begin to prioritise, of course, we do so on the basis of a tacit assumption of the need to strike a compromise between what is ideal and what is practical.These sorts of reflections remind us again that engaged research should be understood as an approach and a process, with engagement taking place on a continuum, rather than as a description of a concrete series of deliverables and outcomes.Instead of looking for operational responses to all these difficulties, it might also help to address the researcher's expectations and assumptions, and to broaden our ambitions and sense of what constitutes 'impactful' or successful engaged evaluative research.Whatever of the 'pure' research outcomes a community-engaged project delivers, a disposition towards engagement, collaboration and mutual respect broadens the mind and guards against complacency.