Evidence to Impact

A community knowledge mobilisation evaluation framework

S. Kathleen Worton
Colleen Loomis
S. Mark Pancer
Geoffrey Nelson
Wilfrid Laurier University

Ray DeV. Peters
Queen’s University

Gateways: International Journal of Community Research and Engagement
Vol 10 (2017)

© 2017 by SK Worton, C Loomis, SM Pancer, G Nelson & R DeV. Peters. This is an Open Access article distributed under the terms of the Creative Commons Attribution 4.0 Unported (CC BY 4.0) License (https://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercial, provided the original work is properly cited and states its license.

Citation: Worton, SK, Loomis, C, Pancer, SM, Nelson, G & Peters, R DeV. 2017, ‘Evidence to impact: A community knowledge mobilisation evaluation framework’, Gateways: International Journal of Community Research and Engagement, vol. 10, pp. 121–142. doi: 10.5130/ijcre.v10i1.5202

Corresponding author: S. Kathleen Worton; kworton@wlu.ca

DOI: http://dx.doi.org/10.5130/ijcre.v10i1.5202

ISSN 1836-3393

Published by UTS ePRESS http://epress.lib.uts.edu.au/journals/index.php/ijcre/index

Within community-based prevention initiatives, there is often a disconnection between research and practice (Waddell 2001). Although extensive effort and substantial resources are invested in the development of community-based interventions, the uptake of these interventions in practice has often been minimal and short-lived (Leadbeater 2010). This disconnect between research and practice could be the result of a lack of effective knowledge-sharing activities, narrow definitions of what constitutes uptake and use of research knowledge (Henry & Mark 2003), or a combination of the two.

The process through which evidence-based practices are shared with practitioners who implement them has evolved and has been refined over a number of decades. A growing body of recent literature in this area includes the development of models illustrating the process of moving knowledge into action (Graham et al. 2006), as well as new interactive strategies for sharing knowledge (Wandersman et al. 2008) that address many limitations of passive knowledge-sharing strategies (e.g. reports, lectures). Despite many advances in connecting research and practice, the gap between research and practice persists (Neal et al. 2015). It is becoming increasingly evident that the process of sharing knowledge is complex and idiosyncratic. Effectively sharing knowledge requires different strategies depending on who is sharing the knowledge, what knowledge is being shared, how it is shared, and the purpose for which it is shared (Ward 2016). Certain strategies for sharing knowledge, such as passive information dissemination and the use of audits and feedback, have been deemed inappropriate for community settings (Kothari & Armstrong 2011; Miller & Shinn 2005). This highlights the importance of carefully tailoring knowledge-sharing strategies to the learning needs and goals of those who are interested in using the information to create positive change. This article focuses on the process of knowledge mobilisation in community settings and has three aims:

1. To highlight the importance of understanding and evaluating knowledge mobilisation in community settings
2. To present a framework to evaluate knowledge mobilisation that captures knowledge use for community stakeholders’ goals
3. To refine the framework by applying it.

Before proceeding, it is necessary to briefly explain our choice of terminology. A variety of terms are used in the literature to describe activities that connect ‘knowledge-to-action’ (Graham et al. 2006). Terms such as mobilisation, translation, transfer, dissemination and exchange have different disciplinary origins and can be used to distinguish different approaches (Ottoson 2009). However, as practice in knowledge sharing advances, distinctions between these terms are becoming blurred, with many terms being used interchangeably. In this article, we use knowledge-to-action to refer to the general process of connecting research and practice. In the framework we propose, we have opted to use the term ‘knowledge mobilisation’ to convey a specific process of sharing knowledge that has been co-created by researchers and community stakeholders in one jurisdiction with community stakeholders in other jurisdictions who may benefit from applying this knowledge locally. Knowledge mobilisation has been formally defined as ‘the reciprocal and complementary flow and uptake of research knowledge between researchers, knowledge brokers and knowledge users—both within and beyond academia—in such a way that may benefit users and create positive impacts within Canada and/or internationally’ (Social Sciences and Humanities Research Council 2016).

KNOWLEDGE MOBILISATION IN COMMUNITY SETTINGS

Much existing knowledge-to-action theory and practice has been developed for clinical or policy settings rather than community contexts. Community settings can be conceptualised as organisations that provide services to the public or to specific populations at a local level, are often guided by a board of directors, and engage community members in forming the organisation’s strategic direction (Wilson et al. 2010). Examples include community-based organisations, public and non-profit organisations, and some direct service providers (Wilson et al. 2010). In community contexts, there are many stakeholder groups involved in knowledge mobilisation, including organisational staff, board members, policy-makers or funders, and community members who benefit from the organisation’s services or are engaged as volunteers. Knowledge-to-action theory and practice literature contains a wide variety of strategies, including highly passive strategies and strategies that require extensive stakeholder engagement. There is therefore a need for more extensive research to determine which approaches work best in particular contexts (Walter, Nutley & Davies 2005).

Many knowledge-to-action strategies commonly used in clinical settings (e.g. passive information-sharing, audits and feedback, the voice of opinion leaders, cost analysis) have been applied to community settings despite a lack of evidence regarding their effectiveness in the community context (Kothari & Armstrong 2011; Miller & Shinn 2005). This is likely because of power imbalance between researchers and community/stakeholders (Isenberg et al. 2004) which results in researchers selecting knowledge-to-action strategies based on assumptions that do not necessarily hold true in community settings (Miller & Shinn 2005). Two assumptions appear to be particularly problematic for sharing research evidence with community stakeholders: 1) the assumption that knowledge producers and knowledge users have similar values and approach innovation in the same ways, and 2) the assumption that implementation of an evidence-based program or practice is the end goal for community stakeholders.

The assumption of similar values and approaches to innovation is challenged by evidence suggesting that community stakeholders are often interested in holistic programs and ecological outcomes while researchers are more focused on targeted interventions and individual outcomes (particularly in clinical settings) (Kothari & Armstrong 2011; Weiss, Lillefjell & Magnus 2016). Researchers also tend to hold a ‘pro-innovation bias’ (Miller & Shinn 2005), prioritising newly developed evidence-based programs over existing practices that may appear beneficial but lack evidence. Community stakeholders often prefer to innovate through the evolution of existing programs using local knowledge rather than through implementing external initiatives (Kothari & Armstrong 2011). Differences in approaches to innovation likely stem from differences in what is considered to be evidence. Researchers often define evidence narrowly in terms of empirical research and emphasise rigour, while community stakeholders often define evidence more broadly in ways that include experiential knowledge and practical wisdom, and emphasise practical utility (Bowen & Martens 2005; Kothari & Armstrong 2011; Miller & Shinn 2005). Regarding the second assumption of similar approaches to implementation, researchers often view implementation of evidence-based practice as the ultimate goal of knowledge-to-action efforts, while community stakeholders may prioritise new ideas or changes in thinking (Bowen & Martens 2005). Implementing an innovation in practice is often a considerably more complex task than it appears to be in knowledge-to-action models (Greenhalgh & Wieringa 2011). This is especially true in community settings, where organisations often work collaboratively and are unlikely to move independently to implement a new evidence-based approach without consulting others in their network (Bowen & Martens 2005; Kothari & Armstrong 2011; Weiss, Lillefjell & Magnus 2016). Many evidence-based practices are expensive, and communities may lack the resources to implement a program in its entirety (Miller & Shinn 2005) and instead opt to use the information to shape their thinking or to enhance existing programs.

Sharing knowledge via strategies that are inappropriate for community contexts may perpetuate impressions that research and practice are separate ‘worlds’ and impede meaningful connection and collaboration between researchers and community stakeholders. Effective evaluation of knowledge-sharing activities is essential to better understand how community stakeholders apply information and to continuously improve knowledge-sharing strategies tailored for community settings.

Knowledge-to-action strategies considered to be most appropriate for community settings often involve collaborative efforts to generate and share knowledge. Engaging community stakeholders in community-based participatory research approaches where they are actively involved in the production of knowledge alongside researchers is a means of promoting knowledge sharing (Wilson et al. 2010). Another recommended strategy is to conduct evaluative research on strong grassroots community initiatives. This provides an opportunity to strengthen existing local practices by developing an evidence base and identifying core components of these programs and initiatives that can be adopted by other communities (Miller & Shinn 2005). There is a need for effective knowledge-to-action strategies that can be used to convey these core components across communities to maximise the benefit of existing community-based research and assist communities in learning from one another. Identifying the processes through which information can be shared most effectively among community stakeholders requires the development of knowledge mobilisation strategies tailored to community audiences. In order to understand how to tailor strategies to community audiences, it is necessary to evaluate knowledge mobilisation and refine the strategies employed based on what works and what could be improved.

EVALUATING KNOWLEDGE MOBILISATION

Despite the existence of numerous knowledge-to-action strategies, minimal attention has been directed towards evaluating knowledge mobilisation. Much of the existing research evaluating knowledge-to-action initiatives is focused on implementation and the examination of more advanced phases of implementation to assess fidelity of the program to the original model (Durlak & DuPre 2008). In knowledge mobilisation of evidence-based community initiatives, an overemphasis on implementation can result in a narrow lens through which to examine community uptake and use of the knowledge shared (Henry & Mark 2003) for three main reasons. First, there are many different forms of knowledge use. Knowledge can be used conceptually to change perspectives or ways of thinking, persuasively to influence the thinking and decisions of others (such as funders or local decision-makers), or instrumentally to make tangible changes to practices (Leviton & Hughes 1981). Second, evaluators focused on implementation may be unable to foresee the ways in which community stakeholders may adapt the knowledge to apply it in practice. This tension between adaptation and fidelity is important to recognise, as there is the risk that knowledge shared could be applied in ways that don’t produce the outcomes promised by the program, or worse be misused to the detriment of organisations or citizens (Cousins 2004). One way of addressing the tension around fidelity vs adaptation is to emphasise the importance of fidelity to core components (i.e. ‘key ingredients’) of the evidence-based practice that are considered to be necessary for the success of the innovation (Hawe, Shiell & Riley 2004; Miller & Shinn 2005). However, this approach is not perfect as it raises questions regarding the extent to which it is realistically feasible to identify and validate core components of every program (Miller & Shinn 2005). Third, knowledge utilisation is often a long-term process in which adaptation is considered ‘inevitable’ (Ashley 2009). Adapted knowledge becomes harder to track during evaluation, and the use of knowledge is more difficult to attribute to a specific knowledge mobilisation initiative when it has been adapted (Blake & Ottoson 2009).

To address some of these challenges, we present, in the next section, a new framework designed to facilitate the evaluation of short-term knowledge use in community settings. We have developed this framework to fit a specific niche: the evaluation of short-term knowledge use across communities. This niche exists between the immediate evaluation of mobilisation activities using ratings of satisfaction, perceived relevance and usefulness (Loiselle, Semenic & Côté 2005) and long-term implementation evaluation assessing the planning, implementation and sustainability of new programs or innovations in a single community (e.g. Stetler et al. 2011; Wandersman et al. 2016). Our proposed framework is intended for evaluating the effectiveness of knowledge mobilisation and capturing different forms of knowledge use in the pre-implementation phase. Pre-implementation activities involve early engagement of community members (by researchers engaged in knowledge mobilisation), opportunities to discuss feasibility of the program or initiative in the community setting, and consideration of community readiness in terms of resource reviewing and cost planning (Chamberlain, Brown & Saldana 2011).

COMMUNITY KNOWLEDGE MOBILIZATION EVALUATION FRAMEWORK

Foundation of the Framework

We propose the Community Knowledge Mobilization Evaluation (CKME) Framework as an approach to evaluating knowledge mobilisation that is designed to capture various forms of knowledge use within community settings. Rather than focusing on implementation of a specific program, this framework is intended for knowledge mobilisation efforts undertaken to convey the key components of a community-based initiative in a way that promotes adaptation and uptake by community stakeholders and allows them to use the information in ways that meet their needs. Our approach to evaluating knowledge use positions social impact as the ultimate goal. Social impact is ‘a consequence of a process in which knowledge and expertise circulates to achieve certain goals that are deemed relevant for the development of society’ (Spaapen & van Drooge 2011, p. 212).

As mentioned previously, the various terms used to describe the process of sharing knowledge emerged from different disciplines. These terms represent knowledge-to-action theories that differ based on what information is shared (e.g. policies, programs, research findings) and how it is shared (e.g. communication, marketing, implementation) (Ottoson 2009). The CKME Framework draws upon knowledge-to-action theories from a number of disciplines to inform key evaluation questions. We incorporate questions on the effectiveness of different formats used to convey the information to community stakeholders (e.g. resources, workshops, discussion forums) developed by drawing on knowledge translation theory in medicine and public health (Ottoson 2009). The process of understanding how information is transferred and shared among community stakeholders is based upon knowledge diffusion theory in communication studies (Rogers 1995). We have informed considerations of knowledge non-use and the matter of context in understanding justifications for non-use by drawing upon knowledge utilisation theory in program evaluation (Cousins 2004). The overarching concept of ‘pathways of influence’, in which evidence informs actions intended to facilitate positive social change (Henry & Mark 2003), is also based in knowledge utilisation literature.

Description of the Framework

The pre-application version of the CKME Framework is presented in Figure 1. The purpose of the framework is to help identify questions to guide examination of varying forms of use that link the social impact goals of the original innovation (from which knowledge is being mobilised) with the social impact goals guiding knowledge use by community stakeholders. Rather than focusing on observable outcomes, this framework guides evaluators to examine the process of knowledge sharing and knowledge use. The components of the framework are linked to specific questions that draw information from community stakeholders and are of sufficient breadth to capture varying forms of knowledge use. By framing the evaluation process in terms of alignment between the knowledge mobilisation activities and the social impact goals of information use, the evaluation process remains flexible by capturing varying forms of use (conceptual, persuasive, instrumental).

Figure 1: Early CKME Framework

Although it is possible (and likely beneficial) to take a mixed-methods approach to evaluating knowledge mobilisation, our framework is primarily intended to guide the qualitative component of evaluation designs. Much of the information the framework is designed to gather can be best gained through interviews with stakeholders who have participated in knowledge mobilisation activities. Additionally, the CKME Framework is designed to be used in a one-time follow-up evaluation of the knowledge mobilisation activities three to five months after stakeholders have received the information to allow sufficient time for them to begin incorporating the information into their roles and networks. The three to five month timeframe also provides the flexibility necessary to accommodate the yearly schedule cycles of community organisations to avoid periods when participants may be unavailable (e.g. end of fiscal year, school summer/winter breaks) or to capture periods of program planning or the development of new initiatives (e.g. funding application cycles).

APPLYING THE COMMUNITY KNOWLEDGE MOBILIZATION EVALUATION FRAMEWORK

To demonstrate the use of the CKME Framework and refine the framework further, we applied it to a transnational knowledge mobilisation initiative for an evidence-based early childhood development project called ‘Better Beginnings, Better Futures’ (Better Beginnings), which is an early childhood initiative with the primary goal of promoting the healthy development of children and families in economically disadvantaged communities (Peters 1994); it is designed to (a) prevent developmental problems, and (b) build capacity of parents, families and neighbourhoods to support healthy child development. Since the late 1980s, this government-funded multi-site project in eight communities in Ontario, Canada (Grant & Russell 1990) has taken an innovative approach to mental health promotion, engaging community residents in developing programs for children. Programs must be ecological and holistic, community-driven, integrated with existing community services and universally available to children and families (Grant & Russell 1990; Peters & Russell 1994; Worton et al. 2014). Longitudinal research followed children (aged 4 to 8) and families in 3 of the 8 project sites and found positive effects of participation of children at several points in the life span, including the most recent assessments when they reached Grade 12 (Peters, Bradshaw, et al. 2010; Peters, Nelson et al. 2010).

To share the lessons learned with other communities, researchers engaged in a one-year pan-Canadian knowledge mobilisation plan. The team developed interactive workshops and delivered them in 7 Canadian provinces and 1 of 3 territories (Alberta, Manitoba, New Brunswick, Nova Scotia, Ontario, Québec, Saskatchewan, and the Yukon) as well as a set of resource materials in English and French covering the following topics (available at bbbf.ca) (Hayward et al. 2011): history, program model, research and evaluation, community resident participation, engaging community partners, project organisation and management, and working with government and other funders. Workshops were scheduled at the request of host community organisations and based on the availability of the presenters. In applying the CKME Framework, we describe the short-term evaluation of the Better Beginnings knowledge mobilisation activities designed to examine how communities are utilising the information shared (if at all) and to understand the motivations and contextual factors influencing use. Two primary research questions guided our application of the CKME Framework:

1. To what extent do knowledge mobilisation activities meet the information needs and learning goals of participants? How could the activities be changed to better meet these needs and goals?
2. To what extent is the information presented in the knowledge mobilisation activities being used by participants in their communities 3 to 5 months following the workshop?
   1. What contextual factors influence this use/non-use?
   2. What goals or expected outcomes do participants using the information hope to achieve?

Method

We used a stratified random sampling strategy to select a sub-sample of 5–6 participants from each province/territory to participate. Participants’ primary perspective in attending the workshop (community member/parent, volunteer, employee with an organisation serving children, government policy) was used for stratification. In a rare case where all participants from a city indicated the same primary perspective, we used an intensity sampling method in which workshop hosts (i.e. community leaders/local workshop organisers) were asked to identify specific workshop participants who could provide in-depth information on the topics of interest. In cases where fewer than 5 individuals from a region agreed to participate in interviews, we included each of these participants in the sub-sample. This was the case in 2 provinces. Interviews were conducted with a sub-sample of participants engaged in a larger mixed methods evaluation that included ratings of satisfaction and readiness for implementation. Although French language knowledge mobilisation activities took place in the provinces of Québec and New Brunswick, the present study includes only stakeholders participating in the English language activities due to limited resources available for translation of the interview materials and limited capacity to conduct interviews in French.

Thirty-one individuals participated in this study. Eight participants attended a workshop in Ontario, 5 in Alberta, 3 in Manitoba, 4 in Nova Scotia, 6 in Saskatchewan, and 5 in the Yukon. Of these participants, 19 identified their primary perspective as employees of an organisation serving children, 6 as government/policy, 1 as a volunteer with an organisation serving children, 1 as having multiple primary perspectives (employees and government/policy) and 4 opted not to specify. Although community members/parents were invited to knowledge mobilisation workshops, no participants in this study identified their primary perspective as a community member/parent. Two participants identified themselves as male, and 29 identified as female.

Qualitative phone interviews (approximately 30 minutes) were conducted with workshop participants using a semi-structured guide with questions about learning goals, information gained and information used. Interviews were digitally recorded and transcribed for coding. Data analysis was guided by the CKME Framework and consisted of five steps: 1) reviewing transcripts to identify key concepts, 2) forming an initial coding scheme, 3) developing thematic codes that encompassed multiple concepts, 4) reviewing themes by conceptual linkages and hierarchies, 5) creating definitions for each theme (Braun & Clarke, 2006; Hsieh & Shannon 2005). Rigour and trustworthiness of findings were enhanced through an audit trail of the primary researcher’s observational, methodological and theoretical memos (Amis & Silk 2008).

FINDINGS

The findings of the evaluation are presented in the CKME Framework’s four main sections: learning goals, knowledge use, process of knowledge use, and outcomes of knowledge use. The first findings section addresses the first research question regarding the effectiveness of knowledge mobilisation in meeting participants’ learning goals. The next three sections address the second research questions regarding knowledge use, contextual factors influencing use (or non-use), and participants’ intended outcomes for use.

Assessing Learning Goals and Information Needs

Overall, the knowledge mobilisation activities offered met the knowledge goals of participants. Most participants approached the workshop and resources to gain knowledge of the initiative, the core principles and unique aspects of its approach, and the best practice findings. Interacting with other attendees to discuss relevance and to build connections also was important. Participants who were building or enhancing existing local initiatives prior to attending the workshop sought tangible strategies for building engagement/partnerships and procedural information. Participants generally indicated that the workshop and resources met their learning needs by providing a comprehensive overview of the program and incorporating sufficient detail, commenting that the information shared was high quality and well presented.

I think the presentation was really quite good. It was in very clear language, very easy to understand, very easy to follow, and gave great examples of how this has been such a success story. The videos that had interviews [with staff] who were actually running a Better Beginnings program were really quite good.

Some participants stated that the workshop provided a valuable starting point in sparking discussion regarding how Better Beginnings might fit with their local communities. Many participants indicated that more practical information on applying the information would be a valuable addition and desired more time for discussion and interaction with other attendees.

The one-day training was good in providing that really good snapshot … It gave us a big overview, but didn’t allow enough time for us to really dig into each section and figure out what exactly worked, what didn’t, and what were the strategies that were really successful.

Assessing Knowledge Use and Non-use

Most participants interviewed indicated that they had used the information from the workshop and/or the resources; some participants noted not having had opportunities for use prior to the interview. Key reasons for non-use appeared to be a perceived lack of organisational readiness, or limits of one’s role, or reticence of organisational hierarchies to engage in discussions about new initiatives and system changes.

The workshop gave me very good grounding … and should we get to the position where [we] were actually working towards achieving something similar I think it would be very helpful at that point, but we aren’t at that position at the moment.

The principles … fit really well with two major large scale research projects that are going on in our province … but who moves ahead with that really gets determined by the superintendent.

Many participants had used and shared the information to reinforce some of the values and approaches already present in their work. Participants’ experiences in using knowledge gained from the knowledge mobilisation activities illustrated a combination of many forms of use. One form of use was to inform new perspectives. Many participants indicated that the initiative’s principles reflected what they were already doing in some of their existing programs (e.g. 50 per cent resident participation in program committees).

I think some of the information in here has helped frame that conversation about why it’s important to have community residents share equal partnership or have equal roles within a coalition. There is that tendency to think that service providers are the experts and community residents are kind of there to learn from the service providers when in actuality I think that it’s the opposite. So getting that 50/50 partnership … that is an ongoing conversation.

Resident participation had previously been a missing component for some participants.

We already have a whole array of programs and services in [our province], and a lot of them are aligned and centralized in one area. What we wanted to take away from this was, how do we connect what we already have to community members and parents and give them a voice, and so we really used the information that was developed through Better Beginnings to integrate into our programs, what we were already doing with communities.

The core components of Better Beginnings were also used for comparing existing practices and identifying new strategic directions.

We used it to evaluate our own practice and then to say, ‘what are we already doing?’ and ‘where are our gaps?’ to evaluate where we could make improvement.

For some, the resources provided a starting point for discussion to reflect on the fit between Better Beginnings and community needs and the possibility of developing a similar initiative.

[The Toolkit] was a discussion starter for those who were coming to the workshop. They were able to then decide whether it’s a good fit for the neighborhoods and that gave us a population of people to speak with or to connect with or to follow up with. It gave us the tools to understand what resources we would need, what kind of support we would need – political support, financial support — and what the role would be of the community organization if we were to move ahead.

Another form of use was acquiring new resources and implementing new practices. A number of participants described using the knowledge to find resources needed to implement new programs.

I used [the resources] to help with a number of grant proposals ... to be able to have additional empirical … and theoretical support. Now we have more fire power, more tools in our tool kit more backup when we’re saying ‘we need this’ because we actually have really good reasons and we have really good documentation. The grant review that I’m sitting for next week is a $20,000 neighborhood grant and it’s my opinion that the research that I was using from Better Beginnings is part of the reason why.

Another participant was involved in efforts to implement the initiative, but the community was still in the process of seeking resources to support it.

We are planning on implementing Better Beginnings, Better Futures. Chats with politicians and forming a steering committee made up of 50% community members are the next steps. [A challenge is] funding – stabilized funding. Whenever we talk to anybody about implementation of any program, it’s ‘oh that sounds great, where are we going to get the money for it?’ Well there is no money so the discussion ends pretty quickly … I’ve had a chance to touch base with MPPs, Councilors and school board super-intendants. They listened and said, ‘oh that sounds like a good idea, good luck!’

In another community, the availability of resources and support of political stakeholders facilitated faster implementation of the program model in a community resource centre initiative.

We basically used Better Beginnings as our model as we pushed forward. After we attended the workshop we were really excited and fired up that this … was the model that we’ve been looking for. Everything just fell right into place perfectly for us…We worked really hard and we managed to get individuals with[in] the city to really buy-in … The two tools that we used were the … summary and the video. The video is a wonderful tool. We’ve used that in numerous presentations and every time we send out funding requests we send that link along so that people have an idea of where it is we want to go.

These findings demonstrate that participants use the information in various ways, including sharing information and starting discussions within their organisations or networks, enhancing existing programs, or starting a new initiative using Better Beginnings as a model.

Early Outcomes and Social Impact Goals Guiding Use

Most participants indicated it was too early to identify tangible outcomes from their use of the Better Beginnings information. It was clear that participants using the information had been able to accomplish outputs such as establishing new relationships with stakeholders (e.g. local politicians), acquiring resources or taking steps towards acquiring resources for new community initiatives, and generating meaningful discussion and new strategies for engaging parents in developing community programs for children.

We had good partnerships with professionals and with schools, but we did not have partnerships with parents as well-developed as we do now.

Parents are now attending parent-child programs whereas before they were always cancelled.

Participants who had used the information applied it for one or more of the following long-term goals: increased resident/parent engagement in programs for children and families, improvement of early childhood intervention services/supports, promotion of children’s health and wellbeing, and increasing the capacity of families to support healthy child development. Many of these goals were central to participants’ work and were being advanced through multiple community initiatives, including community coalitions, participatory research, new programs for parents and families, prevention promotion, and the development of community resource centres for children and families. Participants’ information use supported initiatives with evidence, acting as an established example of core components necessary for achieving program goals. Communities used the information on core components to identify gaps, and strategies to fill those gaps, as well as to draw upon the evidence base and established long-term impact of Better Beginnings to advance new programs or initiatives that used similar approaches.

The whole thing is a community development process, it’s all slow, and it all means you have to take the lead from those who do it not just those of us who have our hands on the pulse of what is new and exciting and the best practices.

I’m looking to build family capacity in addition to engaging the community so that families are less insular and reliant on poor practices and they adopt healthier practices. Ultimately it would be to make sure we’re building engagement with other communities and agencies. We are very inter-dependent and we have to acknowledge that.

Two communities were implementing the Better Beginnings model. One community indicated they were working to implement the full Better Beginnings initiative. The other was actively using the model to guide efforts to develop a ‘hub’ for children and family services after identifying gaps in their existing services. Participants in other communities were actively using the information shared to make changes in local services and programming to advance the Better Beginnings’ goal of promoting the healthy development of children and families in economically disadvantaged communities through: 1) the prevention of developmental problems, and 2) by enhancing the capacity of parents, families and neighbourhoods to support the healthy development of children (Peters 1994). Notably, many of these goals were central to the overarching work of the participants and extended beyond the utilisation of knowledge generated from Better Beginnings. However, each of the participants who used the information used it to enhance the overall goal of early childhood development through resident engagement, partnerships among stakeholders, connections between communities and schools, with a focus on prevention of mental health issues, and development of new programs for children and parents.

In summary, the application of the CKME Framework to Better Beginnings knowledge mobilisation activities led to participants’ learning needs being met, with the exception of the need for adequate discussion and interaction time amongst participants. This knowledge changed perspectives on existing community services, increased understanding of the key components of the Better Beginnings initiative, helped participants identify gaps in community programs and services, informed steps towards adopting missing components, and facilitated acquisition of (or advocacy for) resources and support for new initiatives or practices (e.g. grants and/or generating political support). Instances of non-use among participants were minimal, occurring for those who did not have decision-making authority or those in communities lacking capacity. Overall, the Better Beginnings knowledge mobilisation activities were valuable for community stakeholders, and many used the information gained to inform and advance existing community initiatives aimed at improving services for, and promoting the wellbeing of, children and families.

FRAMEWORK REVISIONS

Although the original CKME Framework worked well for the evaluation presented above, our application informed the need for further development of the framework to better capture knowledge use (Figure 2). The overarching linear pathway between knowledge mobilisation activities and social impact outlined in the early framework was not completely conducive to the flexible process of knowledge use, in which stakeholders use information as a tool to advance existing community initiatives rather than as a catalyst for the development of brand new initiatives. For this reason, we have changed the model to highlight the importance of the link between the social change goals of the original innovation or evidence-based practice (and subsequent knowledge mobilisation activities) and the broad social impact goals of the community stakeholders. This change shifts the emphasis from brand new community initiatives and the assumption of knowledge mobilisation as a catalyst to focus on the alignment of goals between the evidence-based practice and the overarching goals of the stakeholders using the information. We also modified the presentation of the forms of knowledge use by replacing technical (i.e. conceptual) terms with a multi-dimensional component depicting specific knowledge use activities. The cyclical nature of the new knowledge use component is intended to capture the finding that stakeholders are often engaged in multiple forms of use simultaneously, often for the purpose of achieving a single outcome. To encourage greater depth in participants’ responses, we have revised and clarified the wording of some of the questions included in the CKME Framework.

Figure 2: The Community Knowledge Mobilization Evaluation (CKME) Framework

CONTRIBUTIONS OF THE CKME FRAMEWORK

The CKME Framework is a useful tool to assist researchers in identifying pathways between knowledge mobilisation activities and participants’ application of the knowledge to create positive social change for the populations they serve. Researchers engaged in collaborative work with communities require strategies for evaluating knowledge mobilisation to inform what works in what context (Walter, Nutley & Davies 2005). It is important that these evaluation strategies be functional and useful in situations where resources for evaluating knowledge mobilisation are limited. Some existing approaches to evaluating knowledge-to-action strategies focus on the effectiveness of the strategies in producing a specific behaviour change among knowledge users. These approaches to evaluation are more common in clinical settings than in community settings (Kothari & Armstrong 2011) and make use of resource intensive research designs, such as randomised control trials and comparison groups (Bhattacharyya, Estey & Zwarenstein 2011) to identify best practice. This approach to evaluation, although appropriate in some settings, is a poor fit for knowledge mobilisation involving community stakeholders, who may use the information in unpredictable or less obvious ways. In this article, we have focused the development of our framework on evaluation of the short-term impact of knowledge mobilisation activities on advancing community-driven social change initiatives that align with the goals of the evidence-based practice being shared. Our main objective in designing the framework was to create a tool for researchers that would be useful in identifying the early outcomes of knowledge mobilisation activities and capture how community stakeholders choose to apply knowledge given their local context and goals.

This approach to evaluating knowledge mobilisation has some limitations. Asking individual stakeholders about their use of knowledge often fails to capture knowledge applied at the organisational or community level; nonetheless, the approach provides insights into organisational actions. The use of self-reporting methods introduces potential bias, e.g. offering socially desirable responses (Bhattacharyya, Estey & Zwarenstein 2011). Despite its limitations, the approach we propose has important benefits as a tool for short-term evaluations of knowledge mobilisation activities aimed at sharing evidence-based practices generated through community-engaged research and action. The main advantages of the CKME include its simplicity and utility, and its flexibility to capture multiple forms of knowledge use in context.

Simplicity and Utility

The CKME Framework is designed to be short term and pragmatic, and to require minimal resources. The simplicity of the format makes it effective for evaluating broad knowledge mobilisation efforts that aim to share information with a wide range of community stakeholders. This approach to evaluation works well for a knowledge mobilisation initiative like Better Beginnings, where knowledge mobilisation is conducted for the purposes of ‘scaling out’ (sharing knowledge widely across community stakeholders) (Moore & Westley 2011) and introducing new concepts to settings where professional relationships or connections between the researchers and communities are not yet established. The utility of the framework was an important consideration in its design as researchers engaged in community knowledge mobilisation efforts may require an evaluation strategy that captures early outcomes and allows them to meet short-term evaluation requirements of the funders supporting the knowledge mobilisation activities. Although the framework has been designed to be used in the short term, it can provide a basis on which to further engage in additional knowledge mobilisation activities or identify alternative audiences. In the case of the Better Beginnings evaluation, researchers could follow up and build relationships with the communities engaged in implementing the program to help them maintain core components of the program and alignment with program theory as they adapt the program locally (Lee, Altschul & Mowbray 2008). Researchers could also follow up on barriers to knowledge use by ‘scaling up’ and intentionally engaging stakeholders who hold decision-making roles and have the capacity and political influence to create change (Moore & Westley 2011).

Flexibility to Capture Multiple Forms of Knowledge Use in Context

Capturing varying forms of knowledge use is central to the applicability of the framework in community settings. Participants working in communities are likely to use the information gained in varying ways, some of which impact community services directly and others that influence services indirectly. The application of the CKME Framework to the Better Beginnings knowledge mobilisation activities demonstrated the importance of taking a broad approach to evaluating and assessing use. If the evaluation had focused only on implementation of Better Beginnings, results would have shown minimal evidence of uptake during the time period. The approach outlined in the framework has advantages over evaluation strategies that involve assessing only participant satisfaction or perceived usefulness (Chambers et al. 2011). Assessing both the extent to which the activities met participants’ learning goals and the ways in which participants used the information provides insight into individual-level outcomes (e.g. participant satisfaction and changes to attitudes or knowledge) and into some organisational outcomes (e.g. using the information to acquire resources for new programs) or community outcomes (e.g. enhanced relationships and networks among stakeholders). Examining contextual factors underlying instances of non-use (e.g. competing priorities, organisational structure) can inform decisions regarding additional knowledge mobilisation efforts in these communities and inform future knowledge mobilisation strategies.

Long-term Implications

Use of the CKME Framework has long-term implications for academics and community stakeholders as it facilitates the improvement of knowledge mobilisation and supports the transition to full implementation and sustainability of evidence-based practice in community settings.

Continuing evaluation informs ongoing improvement of knowledge mobilisation strategies and the fit of particular strategies with the community context. Careful examination of fit is valuable in ensuring knowledge mobilisation activities are an effective use of time and resources for both community stakeholders and academic researchers. Numerous new strategies for knowledge mobilisation have emerged with advancements in digital communication and social networking (e.g. online workspaces or portals, web conferencing, podcasts) and ongoing evaluation is needed to examine the value of different strategies in different contexts. Findings regarding the effectiveness of knowledge mobilisation strategies in community settings (gained from application of the CKME Framework) can be used to inform the training of researchers as well as the hiring, education and professional development of knowledge mobilisation officers and scholarly communications officers at academic institutions.

As seen in the case study presented, communities engaged in knowledge mobilisation activities can draw upon the evidence base for innovative programs/practices to advocate for support and resources to implement programs/practices or to adapt existing community programs by incorporating evidence-informed components. The CKME Framework is helpful in capturing varying forms of use including (but not limited to) instrumental use leading to full implementation. Use of the CKME Framework can help researchers identify communities intending to progress from the pre-implementation phase to more advanced stages of implementation of an evidence-based practice. Researchers can then draw upon frameworks in the implementation evaluation literature to support communities in undertaking longer term consideration of program outcomes, fidelity and sustainability.

Future Research

Further application of the CKME Framework to the mobilisation of other community-based initiatives is necessary to inform and refine future iterations of the approach. Although the application of the framework to Better Beginnings is informative, application to other knowledge mobilisation initiatives for community-based practices may produce different findings. Better Beginnings is an initiative that was developed and evaluated over a 20-year timeframe. Knowledge about Better Beginnings has been published and shared at various stages of the project through traditional academic avenues (e.g. monographs, journals, conferences) and Better Beginnings programs developed as a part of the demonstration project still run in a number of communities and receive provincial funding. Existing awareness of the principles of Better Beginnings may have set the stage for faster and more extensive knowledge uptake and use.

The Better Beginnings knowledge mobilisation activities aimed to provide many different communities and community organisations across Canada with an introduction to the Better Beginnings approach. This pan-Canadian knowledge mobilisation strategy prioritised breadth in engaging many communities for a short period of time. Future research on the application of the framework to a knowledge mobilisation initiative that is locally focused and prioritises depth of information and long-term engagement would allow for further exploration of the utility of the framework. This application would provide the opportunity for deeper exploration of instances of use and non-use in the pre-implementation phases of evidence-based practice implementation, and could allow for the triangulation of responses across different stakeholder groups and for deeper exploration of social impact over time. Another valuable direction for future research is the adaptation of the framework to examine knowledge mobilisation at baseline. Adapting the framework to gather information on context, learning goals and initial learning outcomes prior to or immediately following knowledge mobilisation activities could enhance existing baseline evaluation approaches that focus on participant satisfaction.

CONCLUSIONS

In this study, we developed the Community Knowledge Mobilization Evaluation Framework and applied it to establish how the framework could help illustrate the effectiveness and impact of knowledge mobilisation activities. In conducting the evaluation, we were able to document the numerous forms of use of the information shared and better understand the strengths of knowledge mobilisation activities and identify ways to improve these activities in the future. These positive results are encouraging for researchers and can help to enhance awareness of the importance of evaluating knowledge mobilisation activities, as well as further improve ways that evidence developed through community-based research activities can be shared for maximum social impact. The framework is a step towards addressing the gap in literature examining the fit between knowledge mobilisation strategies and the information needs of community stakeholders. The CKME Framework captures how community stakeholders use evidence to advance social change goals and develop new local practices and programs.

REFERENCES

Amis, J & Silk, M 2008, ‘The philosophy and politics of quality in qualitative organizational research’, Organizational Research Methods, vol. 11, no. 3, pp. 456–80. doi: d10.1177/1094428107300341

Ashley, S 2009, ‘Innovation diffusion: Implications for evaluation’, New Directions for Evaluation, vol. 124, pp. 35–45. doi: https://doi.org/10.1002/ev.312

Bhattacharyya, O, Estey, E & Zwarenstein, M 2011, ‘Methodologies to evaluate the effectiveness of knowledge translation interventions: A primer for researchers and health care managers’, Journal of Clinical Epidemiology, vol. 64, no. 1, pp. 32–40. doi: https://doi.org/10.1016/j.jclinepi.2010.02.022

Blake, S & Ottoson, J 2009, ‘Knowledge utilization: Implications for evaluation’, New Directions for Evaluation, vol. 124, pp. 21–34. doi: https://doi.org/10.1002/ev.311

Bowen, S & Martens, P 2005, ‘Demystifying knowledge translation: Learning from the community’, Journal of Health Services Research & Policy, vol. 10, no. 4, pp. 203–11. doi: https://doi.org/10.1258/135581905774414213

Braun, V & Clarke, V 2006, ‘Using thematic analysis in psychology’, Qualitative Research in Psychology, vol. 3, pp. 77-101. doi: https://doi.org/10.1191/1478088706qp063oa

Chamberlain, P, Brown, C & Saldana, L 2011, ‘Observational measure of implementation progress in community based settings: The Stages of Implementation Completion (SIC)’, Implementation Science, vol. 6, no. 116, pp. 1–8. doi: https://doi.org/10.1186/1748-5908-6-116

Chambers, D, Wilson, P, Thompson, C, Hanbury, A, Farley, K & Light, K 2011, ‘Maximizing the impact of systematic reviews in health care decision making: A systematic scoping review of knowledge-translation resources’, Milbank Quarterly, vol. 89, no. 1, pp. 131–56. doi: https://doi.org/10.1111/j.1468-0009.2011.00622.x

Cousins, J 2004, ‘Commentary: Minimizing evaluation misuse as principled practice’, The American Journal of Evaluation, vol. 25, no. 3, pp. 393–99. doi: https://doi.org/10.1016/j.ameval.2004.05.005

Durlak, J & DuPre, E 2008, ‘Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation’, American Journal of Community Psychology, vol. 41, no. 3–4, pp. 327–50. doi: https://doi.org/10.1007/s10464-008-9165-0

Graham, I, Logan, J, Harrison, M, Straus, S, Tetroe, J, Caswell, W & Robinson, N 2006, ‘Lost in translation? Time for a map?’, Journal of Continuing Education in the Health Professions, vol. 26, no. 1, pp. 13–24. doi: https://doi.org/10.1002/chp.47

Grant, N & Russell, C 1990, Better Beginnings, Better Futures: An integrative model of primary prevention of emotional and behavioural problems, Ministry of Community and Social Services, Toronto, ON.

Greenhalgh, T & Wieringa, S 2011, ‘Is it time to drop the “knowledge translation” metaphor? A critical literature review’, Journal of the Royal Society of Medicine, vol. 104, no. 12, pp. 501–9. doi: https://doi.org/10.1258/jrsm.2011.110285

Hawe, P, Shiell, A & Riley, T 2004, ‘Complex interventions: How ‘out of control’ can a randomized controlled trial be?’, British Medical Journal, vol. 328, no. 7455, pp. 1561–63. doi: https://doi.org/10.1136/bmj.328.7455.1561

Hayward, K, Loomis, C, Nelson, G, Pancer, M & Peters, R 2011, A toolkit for building better beginnings and better futures, Better Beginnings, Better Futures Research Coordination Unit, Kingston, ON, http://bbbf.ca/ToolkitVideosPDFs/tabid/628/language/en-US/Default.aspx.

Henry, G & Mark, M 2003, ‘Beyond use: Understanding evaluation’s influence on attitudes and actions’, American Journal of Evaluation, vol. 24, no. 3, pp. 293–314. doi: https://doi.org/10.1016/S1098-2140(03)00056-0

Hsieh, H & Shannon, S 2005, ‘Three approaches to qualitative content analysis’, Qualitative Health Research, vol. 15, no. 9, pp. 1277–88. doi: https://doi.org/10.1177/1049732305276687

Isenberg, D, Loomis, C, Humphreys, K & Maton, K 2004, ‘Self help research: Issues of power sharing’, in L Jason, C Keys, Y Suarez-Balcazar, R Tayor, M Davis, J Durlak & D Isenberg (eds), Participatory community research: Theories and methods in action, American -Psychological Association, Washington, DC, pp. 123–37. doi: https://doi.org/10.1037/10726-007

Kothari, A & Armstrong, R 2011, ‘Community-based knowledge translation: Unexplored opportunities’, Implementation Science, vol. 6, no. 59, pp. 1–6. doi: https://doi.org/10.1186/1748-5908-6-59

Leadbeater, B 2010, ‘The fickle fates of push and pull in the dissemination of mental health programs for children’, Canadian Psychology/Psychologie canadienne, vol. 51, no. 4, pp. 221–30. doi: https://doi.org/10.1037/a0020486

Lee, S, Altschul, I & Mowbray, C 2008, ‘Using planned adaptation to implement evidence-based programs with new populations’, American Journal of Community Psychology, vol. 41, no. 3–4, pp. 290–303. doi: https://doi.org/10.1007/s10464-008-9160-5

Leviton, L & Hughes, E 1981, ‘Research on the utilization of evaluations: A review and synthesis’, Evaluation Review, vol. 5, no. 4, pp. 525–48. doi: https://doi.org/10.1177/0193841X8100500405

Loiselle, C, Semenic, S & Côté, B 2005, ‘Sharing empirical knowledge to improve breastfeeding promotion and support: Description of a research dissemination project’, Worldviews on Evidence-Based Nursing, vol. 2, no. 1, pp. 25–32. doi: https://doi.org/10.1111/j.1524-475X.2005.04059.x

Miller, R & Shinn, M 2005, ‘Learning from communities: Overcoming difficulties in dissemination of prevention and promotion efforts’, American Journal of Community Psychology, vol. 35, no. 3–4, pp. 169–83. doi: https://doi.org/10.1007/s10464-005-3395-1

Moore, M-L & Westley, F 2011, ‘Surmountable chasms: Networks and social innovation for resilient systems’, Ecology and Society, vol. 16, no. 1, p. 5, www.ecologyandsociety.org/vol16/iss1/art5/. doi: https://doi.org/10.5751/ES-03812-160105

Neal, Z, Neal, J, Lawlor, J & Mills, K 2015, ‘Small worlds or worlds apart? Using network theory to understand the research-practice gap’, Psychosocial Intervention, vol. 24, no. 3, pp. 177–84. doi: https://doi.org/10.1016/j.psi.2015.07.006

Ottoson, J 2009, ‘Knowledge-for-action theories in evaluation: Knowledge utilization, diffusion, implementation, transfer, and translation’, New Directions for Evaluation, vol. 2009, no. 124, pp. 7–20. doi: https://doi.org/10.1002/ev.310

Peters, RDeV 1994, ‘Better Beginnings, Better Futures: A Community-Based Approach to Primary Prevention’, Canadian Journal of Community Mental Health, vol. 13, no. 2, pp. 183–88.

Peters, RDeV, Bradshaw, A, Petrunka, K, Nelson, G, Herry, Y, Craig, W, Arnold, R & Rossiter, M 2010, ‘The “Better Beginnings, Better Futures” ecological, community-based early childhood prevention project: Findings from Grade 3 to Grade 9’, Monographs of the Society for Research in Child Development, vol. 75, no. 3, pp. 1–176.

Peters, RDeV, Nelson, G, Petrunka, K, Pancer, S, Loomis, C, Hasford, J, Janzen, R, Armstrong, L & Van Andel, A 2010, Investing in our future: Highlights of Better Beginnings research findings at Grade 12, Queen’s University, Kingston, ON, http://bbbf.ca/Portals/15/pdfs/Grade 12 report FINAL version.pdf.

Peters, RDeV & Russell, C 1994, Better Beginnings, Better Futures project: Model, program and research overview, Ministry of Community and Social Services.

Rogers, E 1995, Diffusion of innovations, The Free Press, New York.

Social Sciences and Humanities Research Council 2016, ‘Definitions of terms’, www.sshrc-crsh.gc.ca/funding-financement/programs-programmes/definitions-eng.aspx#km-mc).

Spaapen, J & van Drooge, L 2011, ‘Introducing “productive interactions” in social impact assessment’, Research Evaluation, vol. 20, no. 3, pp. 211–18. doi: https://doi.org/10.3152/095820211X12941371876742

Stetler, C, Damschroder, L, Helfrich, C & Hagedorn, H 2011, ‘A Guide for applying a revised version of the PARIHS framework for implementation’, Implementation Science, vol. 6, no. 99, pp. 1–10. doi: https://doi.org/10.1186/1748-5908-6-99

Waddell, C 2001, ‘So much research evidence, so little dissemination and uptake: Mixing the useful with the pleasing’, Evidence Based Mental Health, vol. 4, issue 1, pp. 3–5. doi: https://doi.org/10.1136/ebmh.4.1.3

Walter, I, Nutley, S & Davies, H 2005, ‘What works to promote evidence-based practice? A cross-sector review’, Evidence & Policy, vol. 1, no. 3, pp. 335–64. doi: https://doi.org/10.1332/1744264054851612

Wandersman, A, Alia, K, Cook, B, Hsu, L & Ramaswamy, R 2016, ‘Evidence-based interventions are necessary but not sufficient for achieving outcomes in each setting in a complex world: Empowerment evaluation, getting to outcomes, and demonstrating accountability’, American Journal of Evaluation, vol. 37, no. 4, pp. 544–61. doi: https://doi.org/10.1177/1098214016660613

Wandersman, A, Duffy, J, Flaspohler, P, Noonan, R, Lubell, K, Stillman, L, Blachman, M, Dunville, R & Saul, J 2008, ‘Bridging the gap between prevention research and practice: The interactive systems framework for dissemination and implementation’, American Journal of Community Psychology, vol. 41, no. 3–4, pp. 171–81. doi: https://doi.org/10.1007/s10464-008-9174-z

Ward, V 2016, ‘Why, whose, what and how? A framework for knowledge mobilisers’, Evidence & Policy: A Journal of Research, Debate and Practice. doi: https://doi.org/10.1332/174426416X14634763278725

Weiss, D, Lillefjell, M & Magnus, E 2016, ‘Facilitators for the development and implementation of health promoting policy and programs: A scoping review at the local community level’, BMC Public Health, vol. 16, no. 1, pp. 1–15. doi: https://doi.org/10.1186/s12889-016-2811-9

Wilson, M, Lavis, J, Travers, R & Rourke, S 2010, ‘Community-based knowledge transfer and exchange: Helping community-based organizations link research to action’, Implementation Science, vol. 5, no. 1, pp. 1–33. doi: https://doi.org/10.1186/1748-5908-5-33

Worton, S, Caplan, R, Nelson, G, Pancer, M, Loomis, C, Peters, RDeV & Hayward, K 2014, ‘Better Beginnings, Better Futures: Conceptual model, research, and knowledge transfer of a community-based initiative for children and families’, Psychosocial Intervention, vol. 23, no. 2, pp. 135–43. doi: https://doi.org/10.1016/j.psi.2014.02.001