PRACTITIONER CASE

Adopting results based management in the non-profit sector: Trócaire’s experience

512201.jpg

512202.jpg

512203.jpg

Eoghan McKernan1, Karen Kennedy2, Allison Aldred3

1 Programme Manager, Irish Centre for Fetal and Neonatal Translational Research (INFANT), University College Cork, 37 Cairn Woods, Mallow, Co. Cork, Ireland

2 Head of Strategy & Impact, Trócaire, Maynooth, Co. Kildare, Ireland

3 CEO, Suas Educational Development, Floor 1/2 Unit 3, Whitefriars, Aungier Street, Dublin 2, Ireland

Project Management Research and Practice, Vol. 3, July-Dec. 2016

© 2016 by the author(s). This is an Open Access article distributed under the terms of the Creative Commons Attribution 4.0 International (CC BY 4.0) License (https://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license.

Citation: McKernan, E., Kennedy, K. & Aldred, A. 2016. Adopting Results Based Management in the Non-Profit Sector: Trócaire’s Experience. Project Management Research and Practice, 3, 5122. http://dx.doi.org/10.5130/pmrp.v3i0.5122

Published by UTS ePRESS | http://pmrp.epress.lib.uts.edu.au

Corresponding author: Eoghan McKernan, Programme Manager, Irish Centre for Fetal and Neonatal Translational Research (INFANT), University College Cork, 37 Cairn Woods, Mallow, Co. Cork, Ireland. eoghan.mckernan@ucc.ie

DOI: http://dx.doi.org/10.5130/pmrp.v3i0.5122

SYNOPSIS

The increasing emphasis on aid effectiveness, accountability, and impact measurement in international development and humanitarian work has generated a requirement for high-quality internal systems for managing programs. To help to address this requirement, Trócaire adopted results based management in the 20 countries in which it works. This article provides an overview of Trócaire’s results based management journey, including the process of embedding the new approach in the organization, the lessons learnt from this process, the subsequent benefits that are emerging at field program level, and the challenges going forward.

PROGRAM SETTING

Trócaire began moving toward results based management in 2009. However, from 2011, partially in response to increased internal and external pressure, results based management was rolled out, focusing initially on Trócaire’s international development work. This involved a revision of program management processes, in particular around monitoring and evaluation, and program approval. There was an extensive period of consultation and awareness-raising, to address concerns regarding the implications of what was initially seen as a “donor driven” agenda. This was supplemented by dedicated staff training and coaching, which focused on both the principles and practical aspects of results based management.

BUSINESS NEED

Trócaire is the official overseas development agency of the Catholic Church in Ireland. Established in 1973, Trócaire works in over 20 countries today on issues including sustainable livelihoods, governance and human rights, gender equality, HIV, climate change, and emergency relief. As an organization that directly employs 380 staff and spent almost €58 million in 2014–15, helping to improve the lives of an estimated 2.4 million people directly and 19 million people indirectly in Africa, Latin America, the Middle East, and Asia, Trócaire makes an interesting case study into the use of results based management to enhance project management.

TARGET READER

Project managers in the nonprofit sector, practitioners in international development and humanitarian sectors, and project managers and practitioners in the public and for-profit sectors

YEARS OF PROJECT

2009–14

COMPETENCIES HIGHLIGHTED

Improved practice in program design; planning, monitoring, evaluation, and management and reporting; greater accountability; robust evidence of impact
RELATED THEORY
Management by objectives, management for results

LESSONS LEARNED

Results based management has become firmly embedded within the organization. While challenges remain, feedback from staff shows appreciation of the benefits the approach can bring. Results based management has provided significant learning for staff and local partners, and is yielding positive results in terms of clearer logic in the aims and objectives of programs; better and more nuanced design of programs, including the opportunity to strengthen gender mainstreaming within programs; better managed programs; higher-quality annual reports; and more robust evidence of change in people’s lives.

Background to the results focus

Trócaire is the official overseas development agency of the Catholic Church in Ireland. Established in 1973, Trócaire works in over 20 countries today on issues including sustainable livelihoods, governance and human rights, gender equality, HIV, climate change, and emergency relief. In Ireland, Trócaire raises awareness about the causes of poverty through education outreach programs, parish networks, and public campaigns and advocacy work. Trócaire’s overseas work is carried out almost exclusively through partners. Trócaire’s partners are local organizations and networks, who work with poor and marginalized people and communities to tackle poverty and injustice.

The increasing emphasis on aid effectiveness, accountability, and impact measurement in the international development and humanitarian sectors has generated a requirement for high-quality internal systems for managing programs.

At the Second High Level Forum on Aid Effectiveness in Paris, France, in 2005, it was recognized that aid could and should be producing better impacts. The Paris Declaration was endorsed in order to firmly base future aid efforts on first-hand experience of what works, and does not work, with aid. The declaration outlines the following five fundamental principles for making aid more effective: (1) ownership by partner countries; (2) alignment with each country’s poverty reduction strategies and systems; (3) harmonization among donor countries; (4) focus on development results and on measuring results; and (5) mutual accountability. The concept of “aid effectiveness” therefore entered development discourse as embodying a package of specific ideas and measures (Hayman 2009).

Under the fourth principle (results), “results based management” and “results oriented reporting and assessment frameworks” are highlighted as a means toward achieving aid effectiveness. In the Paris Declaration, managing for results means managing and implementing in a way that focuses on desired results, and uses information to improve decision-making. In Clause 46, partners and donors jointly commit to “work together in a participatory approach to strengthen country capacities and demand for results based management” (Vahamaki et al. 2011). In 2008, at the Third High Level Forum on Aid Effectiveness held in Accra, Ghana, an even greater number and wider diversity of stakeholders endorsed the Accra Agenda for Action. This agenda both reaffirms commitment to the Paris Declaration and calls for greater partnership between different parties working on aid and development (OECD 2014).

The Open Forum for CSO Development Effectiveness held a global forum in Istanbul, Turkey, in 2010. At the forum, the Istanbul principles for civil society organization (CSO) development effectiveness were established. Principle 8, “Commit to realizing positive sustainable change,” states that:

CSOs are effective as development actors when they […] collaborate to realize sustainable outcomes and impacts of their development actions, focusing on results and conditions for lasting change for people. (Open Forum for CSO Development Effectiveness 2014)

Following the High Level Forum on Aid Effectiveness held in Busan, South Korea, in 2011, the results agenda has received even more emphasis and is currently a top political priority (Vahamaki et al. 2011). There is now a strong focus in the nonprofit sector on results and managing by results to help deliver on aid effectiveness.

Defining results based management

The Organisation for Economic Co-operation and Development (OECD) defines results based management (RBM) as “a broad management strategy aimed at achieving important changes in the way in which agencies operate, with improving performance and achieving results as the central orientation” (Binnendijk 2000). The Department of Foreign Affairs, Trade and Development (DFATD) Canada, defines RBM as a program/project life cycle approach to management that integrates strategy, people, resources, processes, and measurements to improve decision-making, transparency, and accountability. The approach focuses on achieving outcomes, implementing performance measurement, learning, and adapting.

The United States Agency for International Development (USAID) defines a result as a change in the condition of a customer, or a change in the host country condition, which has a relationship to the customer. Results are linked by causal relationships, that is, a result is achieved because related, interdependent results were also achieved (USAID 1999).

Trócaire defines a result as a change happening in the lives of people as a consequence of a project or program. The change may concern the economic, social, political, cultural, or ecological situation of people. Trócaire considers RBM to be a program and project management method that focuses on what a program is achieving, as opposed to what activities are being implemented in the program. Key to RBM are the principles of focusing on change in the target population, learning from results throughout program implementation, and subsequently adapting implementation if necessary.

History of RBM

The roots of RBM date back to the 1950s, when Peter Drucker introduced the concept of management by objectives in his book, The practice of management (Drucker 1954). The private sector was the first to manage by objectives, which later developed into the logical framework for the public sector and was adopted by USAID in the late 1960s (UNESCO 2011).

Throughout the 1990s and 2000s, the Canadian Government and the Canadian International Development Agency (CIDA, now part of DFATD) further developed RBM. The OECD Development Assistance Committee established the Working Party on Aid Effectiveness and Donor Practices, which recognized the importance of RBM in the context of the effectiveness of development cooperation at its first meeting in Paris, in 2003 (Meier 2003).

Over the past 10 to 12 years, due to growing financial constraints and a global debate on the effectiveness of aid, there has been significant external pressure for development cooperation agencies to steer their management systems toward effectiveness. RBM systems have been introduced in most developed country government sectors, while implementation of RBM is ongoing in most developing countries (Vahamaki et al. 2011).

A number of international aid agencies have been using RBM for some time. DFATD in Canada states that it has been using RBM, in one form or another, for over 30 years. CIDA’s first official RBM policy was released in 1996, and a revised and updated policy was approved in 2008 (DFATD Canada 2014). The Danish International Development Agency (Danida) says that using a managing for development results approach helps to ensure that all resources are directed toward achieving set goals. The goals must be clear, concrete, measurable, limited in number, and with a clear timeframe. Monitoring, documenting, and reporting goals are emphasized (Danida 2014).

RBM has been part of the United Nations reform agenda, which seeks to improve coherence of the United Nations system, its effectiveness, and its accountability, for over a decade. There have been increased efforts on the part of the United Nations development system to enhance RBM within individual agencies and at the country level. The application of RBM in the development field has gained currency, and national governments and public institutions increasingly are adopting this approach (Bester 2012). Serrat et al. (2015) report that the Asian Development Bank mandated use of the logical framework for project design and monitoring in 1995, initiated results-based country partnerships strategies in 2005, instituted annual development effectiveness reports in 2007, and cascaded results frameworks into departmental, office, sector, thematic, and individual work plans in 2011. Indeed a case study demonstrates how the Asian Development Bank helped the Government of the Philippines to ramp up RBM, reaching a milestone in 2013 when the Department of Budget and Management refined its Organizational Performance Indicator Framework, revised major final outputs and performance indicators, and restructured programs, projects, and activities toward the FY2014 National Expenditure Program, worth ₱2.268 trillion.

Concern has been expressed by many across the development sector, who believe that the focus on results has narrowed the view of what is valued and how value is measured, and is promoting work that is easy to measure, while often the work that is most transformational is the least measureable (transformational referring to changing power relations and structures that perpetuate inequality and injustice). For example, Hulme (2010) states that RBM encourages a focus on identifying and continuously monitoring SMART (specific, measureable, agreed, realistic, and time-limited) goals, indicators, and targets, and that this focus on measurability has led to a reduced interest in difficult-to-measure goals such as human rights, participation, and democracy.

According to Hatton et al. (2007), many development practitioners view RBM in a negative light, considering it a donor requirement that diverts time, energy, and resources away from development work. However, in reflecting on the challenges associated with RBM, they suggest a number of strategies to improve its use and conclude that development practitioners need to be more aggressive in implementing RBM.

While RBM has both accountability for results and managing for results functions, Ika (2012) considers RBM, currently, to be too heavily focused on accountability for results and not focused enough on managing for results.

According to Eyben et al. (2015), while development organizations are increasingly seeking to understand better what works, many in the sector have felt that results and evidence protocols and practices have constrained their ability to pursue transformational development. However, there is also recognition that the results agenda can create opportunities for people-centered accountability processes, to promote useful debates about value for money, and to provide insights on power dynamics using theory of change approaches.

Vahamaki et al. (2011) conclude that, despite the large amount of literature and research focusing on the difficulties and challenges associated with implementing the results agenda in the development community, there are examples of success and recommend, based on several theories, how to improve RBM as it is currently practiced.

Irish development sector context

RBM has been part of the discourse in the international development sector in Ireland for a number of years. Irish Aid provides multi-annual funding to a number of international nongovernmental organizations (NGOs) through the Irish Aid Programme Grant. In the “Irish Aid Programme Grant 2012–2015 Programme Cycle Management Guidelines” (revised December 2013), Irish Aid stated that a “shared understanding of why managing for development results matters and what is meant by ‘result’ is a critical element to the success of this (programme grant) partnership.” The guidelines state that NGOs receiving funding must demonstrate clarity of understanding of the pathways of change through to the bigger picture; a clear process of analysis and prioritization of results that the organization has chosen to strive for; recognition and development of competence in a given field; and the ability to plan for, monitor, report, learn from, and make evidence-based decisions that contribute to the achievement of results.

Irish NGOs that are members of Dóchas, the Irish Association of Non-Governmental Organisations, have formed a Results Learning Group to engage in shared learning, discussions, and dissemination of best practice in the implementation of an RBM approach (Dóchas 2014). Group members include Trócaire, Concern, Oxfam Ireland, Goal, Suas, Self Help Africa, Christian Aid, Child Fund, Aid Link, and Sight Savers, among others.

Trócaire’s approach to field programs

Over 2013–14, Trócaire implemented 61 programs in 20 countries in Africa, Asia, the Middle East, and Central America. The programs were divided across five thematic areas: Preparing for and Responding to Humanitarian Crises, Governance and Human Rights, Sustainable Livelihoods and Environmental Justice, Promoting Gender Equality, and Addressing HIV and AIDS. The majority of Trócaire’s development programs follow a five-year program cycle.

Table 1 provides an overview of Trócaire’s programs and the estimated number of beneficiaries reached in 2013–14.

Table 1 Overview of Trócaire programs 2013–14
Thematic area Programs Beneficiaries
Preparing for and Responding to Humanitarian Crises 15 1,100,000
Governance and Human Rights 20 1,552,000
Sustainable Livelihoods and Environmental Justice 15 656,000
Promoting Gender Equality 6 78,000
Addressing HIV and AIDS 5 80,000
Totals 61 3,466,000

Almost all of Trócaire’s programs involve a number of partners implementing projects, which collectively form a coherent program. Each program is essentially a set of complementary interventions, implemented at different levels in partnership with several organizations to achieve a common overall objective. The program approach promotes an integrated response. For Trócaire, “mainstreaming” is essential to ensure it does not lose sight of issues which are key to its values and effectiveness. Mainstreaming refers to the strategy of integrating a particular consideration, such as gender, into all aspects of program and policy design, implementation, monitoring, and evaluation.

Trócaire’s interpretation of results: defining and managing by results

Critically, results are different from program and project activities and outputs. An output is the tangible product of an activity. The purpose of the output is to induce a change (a result) in the target population. Consequently, every activity has an output and every output leads to a change (see Figure 1).

512204.jpg

Figure 1 Output versus Result

Organizations like Trócaire and its partners conduct activities and realise outputs. However, it is only when the target population engages with a project that results are achieved, that is, changes happen within the target population.

In RBM, results are the key element of a program. Results, that is, changes, are the focal point of the program: from the identification and design processes right through to implementation, monitoring and evaluation, and reporting of the program.

Managing for results throughout the program cycle

RBM is a project and project management approach, using information on the results a program has (or has not) achieved to guide project or program implementation (see Figure 2). RBM requires the identification of expected results prior to commencing program implementation, and periodically checking to see if those results have been achieved. Based on this comparison, the program implementation may be adapted.

512205.jpg

Figure 2 Managing for results in the program cycle

Consequently, in RBM, results are the reason for developing a program: they inform the implementation and management of a program, they are the targets of a program, and they are used for monitoring and evaluating a program.

Core RBM components and tools

There are four components that are core to the use of RBM at Trócaire. Each of these components is supported by dedicated tools, described below.

  1. The identification of expected results. This is supported by the Logic Model, which outlines the logic of the program by identifying layers of change.
  2. A system to measure results through the identification of indicators, baselines and targets, and a process to monitor the indicators. This component is supported by the Monitoring and Evaluation Framework.
  3. The identification and active management of risks to programs. This component is supported by the Risk Register.
  4. A process of review and adaptation, which provides the opportunity and motivation to reflect on the results achieved and identify what adaptations, if any, should be made in the program. At Trócaire, the tool to support this process is the Annual Programme Review Process.

THE THREE LEVELS OF RESULTS

Trócaire’s interpretation of results has been based on CIDA’s definition of the results chain (DFATD Canada 2011). Trócaire sees a program as a process of social change among women, men, and communities. In order to better understand this process, Trócaire distinguishes three levels or steps of change, based on CIDA’s definition.

  • The first level of change is the immediate outcome. This is the acquisition of new ideas, new awareness, new knowledge, or new skills, or access to new resources.
  • The second level of change is the intermediate outcome. This is the utilization of the newly acquired ideas, awareness, knowledge, or skills, or new resources, which result in new behaviours, attitudes, and practices.
  • The third level of change is the long-term outcome. This is the new situation, or the new sustainable condition in a community as a result of new behaviours, attitudes, and practices.

This process of change is initiated and supported by the activities and outputs of the program. In effect, an activity, output, immediate outcome, intermediate outcome, long-term outcome, and impact form a “chain of causality.”

Table 2 shows the simplified layout of a Trócaire Logic Model. In practice, most of Trócaire’s field programs have two or three long-term outcomes. For each activity there is an output, and for each output there is an immediate outcome (see Figure 3). It may take two or more immediate outcomes to produce an intermediate outcome, and it may take two or more intermediate outcomes to produce a long-term outcome.

Table 2 Example of Outputs leading to Outcomes
Activity Output Immediate outcome Intermediate outcome Long term outcome
To train target men and women in new agricultural techniques Training conducted Women and men have acquired new knowledge in agricultural techniques Women and men in target community practice new agricultural techniques Increased food security for women and men in the target community
To provide new, improved seeds Seeds provided Women and men have acquired a new resource (improved seeds)
To put in place a micro-credit facility Micro-credit facility in place and functional Women and men have access to a new financial resource

512206.jpg

Figure 3 Levels of Results

The journey: the process of adopting RBM at Trócaire

Trócaire recognized that an initial weakness in the implementation of its program work was a lack of systems for setting agendas and measuring progress. In response, a Strategy and Impact Unit was established in 2009, whose mandate included supporting all divisions to put in place systems and structures to help Trócaire plan, manage, and demonstrate the impact of its work.

While it was always organizational policy to report annual progress for each program to head office and donors, and to conduct end-of-program evaluations, until 2009 it was not mandatory to conduct baseline studies at the beginning of each program or to report progress against results-focused outcomes. Rather, most programs reported progress using activity-focused monitoring data, while measuring change in people’s lives was the focus of program evaluations. Between 2009 and 2011, Trócaire’s International Programme Advisory Committee, a subcommittee of the Board, repeatedly found that evidence of impact in Trócaire programs was difficult to verify, as programs at that time lacked the baseline data needed to compare program results.

Trócaire effectively began moving toward RBM in 2009, when it became organizational policy for each field program to conduct a baseline study, to establish the status of the target population prior to program implementation. Field programs also had to report annual progress, as well as mid-term and final program evaluation progress, against the baseline situation. It was also decided to encourage the use of more standardized outcomes and indicators for programs with a similar theme (that is, Preparing for and Responding to Humanitarian Crises, Governance and Human Rights, Sustainable Livelihoods and Environmental Justice, Promoting Gender Equality, and Addressing HIV and AIDS).

INTRODUCTION OF RESULTS FRAMEWORKS TO MANAGE AND REPORT

In 2010, Irish Aid requested reporting by results in the Multi-Annual Programme Scheme annual report and asked Trócaire to submit a number of results frameworks. In response, Trócaire developed an indicative portfolio of results frameworks for 10 programs.

In 2012, Trócaire entered into a funding partnership with Irish Aid under the Irish Aid Programme Grant 2012–2015. In the memorandum of understanding that Trócaire signed with Irish Aid, there are two benchmarks related to RBM.

  • Benchmark 1: embed the RBM approach organizationally, strengthen program logic, and validate its theory of change.
  • Benchmark 2: develop a full baseline and ensure that all [results] frameworks (including development education) will meet the required standard, as stipulated by Irish Aid by the end of 2012.

The “Irish Aid Programme Grant 2012–2015 Programme Cycle Management Guidelines” explained that Irish Aid required all NGOs to submit a results framework (or set of frameworks) for their approved program of work. In addition, it stated that before annual grants are disbursed, adjusted results frameworks must be submitted and approved by Irish Aid.

STRATEGIES USED TO ROLL OUT RBM

Tasked with embedding RBM in the organization, the Strategy and Impact Unit adopted a number of strategies, as described below.

  • In 2010, the indicative portfolio of results frameworks was developed. This would provide the basis of future iterations of results frameworks.
  • The capacity of the Strategy and Impact Unit was increased to a staff of four in 2011.
  • From 2011, every opportunity was taken to raise awareness of and discuss the results agenda at meetings with international management and staff, in Ireland and overseas.
  • New results framework templates were developed in late 2011, and piloted in early 2012. A series of training webinars were conducted for all field-based staff to provide guidance on using the new results frameworks.
  • A two-day feedback workshop for the pilot was conducted at Trócaire’s head office, in April 2012. This provided the basis for refining concepts and formats, and made clear the imperative for rigorous face-to-face RBM training for all program officers and their line managers in the field.
  • A Monitoring and Evaluation Working Group was established following the feedback workshop (with strong field representation), to help refine concepts and formats, and to agree and arrange the rollout of face-to-face RBM training with all relevant colleagues.
  • A rigorous three-day training session in RBM concepts, and in the development and use of results frameworks, was provided to all program officers and their line managers.
  • Intensive collaboration took place between the Strategy and Impact Unit and program teams (thematic advisers) at head office, during the subsequent Annual Programme Review Process and submission of results frameworks to Irish Aid.

KEY SUCCESS FACTORS

Designing and rolling out dedicated training in Ireland and overseas was the single most important strategy, and was key to bringing the organization forward in terms of RBM. During the initial stages, Trócaire brought in a consultant with significant RBM experience to assist. The consultant had a strong approach and a lot of credibility, and delivered robust and exacting training. This was central to its success and helped to change attitudes among colleagues toward RBM.

Three-day regional trainings were conducted in Kenya and Ethiopia for Horn and East Africa staff, in Kigali for Central and West Africa staff, in Harare for Southern Africa staff, and in Cambodia for Asia staff, in mid to late 2012. Training for staff in Central America was held in Nicaragua, in early 2013.

Three-day country-specific trainings, which focused on Trócaire partners, were conducted in Ethiopia, Liberia, Haiti, and Zimbabwe over 2012 and 2013.

Annual submission of results frameworks to Irish Aid was also an opportunity to negotiate and agree Trócaire’s approach to RBM.

Key to the extensive rollout of RBM was “buy-in” from senior management, both in the international division and at director level. Senior management ensured that there was considerable investment in RBM: that resources were provided to support rollout strategies and that the capacity of the Strategy and Impact Unit was strengthened (growing from two staff between 2009 and 2011, to four staff from 2012 onward).

The initial trainings have been complemented, with ongoing mentoring and support delivered as required by colleagues in the field, thematic program advisers at head office, and Strategy and Impact Unit advisers. The initial trainings have also been complemented with “refresher” RBM trainings, conducted at three global meetings in 2013 (in Kenya, Ethiopia, and Malawi) and four global meetings in 2014 (three in Ireland and one in Uganda). These refresher trainings are available to all Trócaire field program officers, and will be delivered by the Strategy and Impact Unit as the opportunity and need arise.

Significant investment in thematic program advisers in RBM approaches – through the development of tools and resources for baselines, monitoring, and evaluations – contributed significantly to the success of the rollout. Joint input from the program and Strategy and Impact Unit teams also helped to ensure that a standard approach to RBM was adopted, as well as tailored to thematic areas.

Further skills development training was conducted in 2013 and 2014, in areas critical to RBM such as program design and program monitoring, through Trócaire’s Programme Cycle Management Development Programme conducted at all global thematic meetings held in 2013 (three-day trainings on program monitoring) and 2014 (three-day trainings on program design).

There has been an ongoing process of streamlining outcomes, and indicators and tools for measurement, to ensure the process of measuring results assists program management, rather than overwhelms the program and thus interferes with implementation.

CHALLENGES

A number of challenges emerged with the rollout of RBM. Some of these have shaped subsequent iterations and technical support, while others are still works in progress.

The time and resources required to roll out RBM were significant across the organization, involving head office-based program advisers, Strategy and Impact Unit teams, and field-based staff and partners. Much learning was taking place while the RBM process was rolled out, leading to constant modifications and dialogue between the field and head office. This was often complicated by the lack of appropriate monitoring tools and skills at partner level. As processes bed down, it has also become apparent that Trócaire and its partners initially collected too much and too varied data against indicators. In later iterations, the number of indicators used to measure change has been reduced to a more manageable level.

A key challenge was to devise appropriate measurements for transformative and long-term change in empowerment and gender equality. A number of participatory tools and innovative methodologies have been developed and are in use, but were not available in the early days of the rollout.

Ideally, all staff would have received intensive face-to-face training in RBM before they were required to adopt results frameworks in 2012. However, the Strategy and Impact Unit did not have the capacity to organize and deliver the training required in advance. The somewhat piecemeal approach to RBM training before 2012 caused some frustration among program officers, who felt they were required to use results frameworks and manage their programs by results without fully understanding the approach. As the program manager in Malawi described it:

[…] bits and pieces of information concerning RBM and results frameworks came to us at different times and it felt a bit disorganized, and it initially wasn’t clear what was required from partners. The dedicated RBM training that the organization rolled out in 2012 and 2013 was very good. Everyone is now at the same level of thinking. We needed them to bring everyone to the same level. But it happened a bit late.

In addition, in most cases program officers were mid-program and retro-fitting their existing program monitoring and evaluation systems into results frameworks. This limited the extent to which staff could completely embrace RBM concepts.

Impact of RBM training/adoption of RBM

In determining where Trócaire is now with RBM, this section is informed mainly by the perspectives of:

  1. the Strategy and Impact Unit, as the unit charged with rolling out RBM;
  2. quality teams within thematic program areas, gathered during Trócaire’s organizational strategic review process;
  3. partners, gathered via a partner survey conducted in July 2014;
  4. staff, gathered via an organization-wide staff survey; and
  5. individual staff members, gathered via interviews addressing the experience of adopting RBM in their programs, the benefits and challenges of using RBM, and the challenges and opportunities of RBM going forward.

All of Trócaire’s development programs and a number of its humanitarian programs now use results framework tools, including the Logic Model, Monitoring and Evaluation Framework, and Risk Register. These tools, or documents, are now a prerequisite to program approval; fundamental to ongoing monitoring and evaluation; and necessary for annual reporting of progress against targets, for communicating 12-month targets, and for highlighting changes to program strategies and end-of-program targets. Using these tools helps to ensure that all development programs are designed to achieve clear results, supported by annual measurement. In turn, this ensures that across all programs Trócaire now has data on:

  • whose lives Trócaire is impacting, and the nature and scale of change achieved versus the changes Trócaire aimed to bring about;
  • where this change is happening;
  • which program interventions are working well, in particular contexts, and which are working less well;
  • how implementation can be improved; and
  • how change can be better measured, that is, which tools work well.

Perspectives from Trócaire staff

As part of an organizational strategic planning process, Trócaire conducted a review of progress over the last five years of its current strategic framework. According to thematic program staff, there has been significantly improved measurement at beneficiary level and much clearer indications of impact in recent years. The review found that a substantial and sustained investment in measuring measure impact – particularly through the rollout of RBM, complemented by investments in program cycle management processes and staff training – has been important in strengthening Trócaire’s ability to deliver program outcomes, in terms of positive change in people’s lives.

Likewise, investments in monitoring and evaluation in other parts of the organization, notably in development education, policy, and campaigns, are cited as contributing to major improvements in this work. Informal feedback from peers in the development education sector in Ireland is that Trócaire’s Development Education Results Framework as an example of best practice.

A staff survey conducted at both field and head office level in July 2014 indicated that training in RBM was well received, and the rollout of RBM has brought in new and better systems.

Perspectives from the field

A number of interviews were conducted with field-based colleagues in preparation for this article, to get some more personal reflections on the process of adopting RBM in their countries and programs. According to Trócaire’s country director in Zimbabwe, the adoption of RBM has been well-received, but requires ongoing support and follow-through to ensure correct use:

Because of RBM, program design is better, and programs are doing better data collection and better design, informed by better analyses. This should lead to more impact for beneficiaries […] once approved, they [logical frameworks] were not looked at again until evaluation time. Logical frameworks were not used as a monitoring tool. Using results frameworks has changed all this.

The gender equality program officer in Pakistan, initially found:

Trócaire’s approach to RBM is complicated, but – in terms of monitoring where we are, proposed changes, etcetera – is useful. Other program management approaches were focused on outputs. RBM is a fair way of evaluating programs and revising strategies. We are much clearer on the change we’re expecting to see, so we’re putting more efforts into seeing these changes.

IMPACT OF RBM APPROACHES ON TARGET POPULATIONS

According to Trócaire’s country director in Zimbabwe:

[…] in a few years, when we have gone through a couple of cycles of managing programs through RBM, we will have really good data and proof of what we’re doing. For now, we have changed our own organization, which has changed how partners operate, which should change the lives of beneficiaries.

According to the program manager in Malawi, there have been benefits to the population in the Sustainable Livelihoods and Environmental Justice, Promoting Gender Equality, and the Addressing HIV and AIDS programs as a result of using RBM to manage them. In the Governance and Human Rights programs, both partners and the target population are actively using monitoring data to make representation to local government.

PARTNERS’ EXPERIENCE OF RBM IN TRÓCAIRE PROGRAMS

Trócaire conducted a partner survey in July 2014 (with a sample of 123 partner organizations). The survey showed that partners highly value Trócaire’s capacity building support, and highlights capacity building on financial management, management skills, technical abilities, and monitoring and evaluation as particularly valuable.

When partners were asked to rate how much they agree or disagree with statements about Trócaire’s monitoring and reporting, the average score was 5.85 out of a possible 7 (responses ranging between “somewhat agree” and “agree”), indicating partners believe that Trócaire engages in good practice when it comes to monitoring and evaluation. Reassuringly, the statement “the monitoring we do for/with Trócaire helps us to improve what we do” attracted the highest average score of 6.12 out of a possible 7 (between “agree” and “strongly agree”), closely followed by “reporting requirements for Trócaire are clear.”

There has been some improvement in partner ratings for monitoring and evaluation support since 2010. According to one partner surveyed in 2014, “The programmatic focus on strengthening monitoring and evaluation, as well as financial planning/skills building, has been extremely useful.”

Almost all partners in Zimbabwe have received formal training in RBM from Trócaire. According to Trócaire’s country director in Zimbabwe, the major shift has been the requirement of partners to report on impact rather than activities. They have struggled with this and found it complex, needing tools and guidelines to assist them. However, partners now have practical and tangible ways to to quantify their impact. Some are surprised about being able to quantify impact in governance programming.

New partners are really taking it [RBM] on. They are impressed at Trócaire’s approach to RBM and see it as an opportunity. There were high levels of interest in the follow-up training that was arranged. Partners wanted other key staff in their organization to go through RBM training. (Trócaire country director, Zimbabwe)

This sentiment is echoed by a partner staff member, who believes that RBM is a good tool:

It guides us on what to monitor and evaluate, and what to change. It gave shape to the project. Before, we had the logical framework, but we didn’t know what we would look for at the end of the day […] When we had the RBM training with Trócaire, we went back, fine-tuned some areas that were loose. It helped us to look at what we were doing. It helped with the development of a new program for Comic Relief. The knowledge we have gained in the last three years was useful for the application.

In Pakistan, partners also received training in RBM, organized by Trócaire. According to the gender equality program officer:

In 2012, partners were at zero RBM level. But now, they are using the Trócaire results framework format. They need support still, but they are clearer. I have seen change of mindset at partner level. They used to report against activities. Now they are thinking about results and changes they have brought in the lives of people. However, partners need diverse support in designing, monitoring, and analyzing data. Someone needs to work with them.

However, partners in many of Trócaire’s program countries have still not been formally trained in RBM.

CURRENT STANDARD OF PROGRAM RESULTS FRAMEWORKS

Compared with 2011, when results frameworks were not standard practice, partners are now much better at demonstrating the use of monitoring information in decision-making and active program management. In Trócaire’s Annual Programme Review Process, each program submits its results framework along with progress made in the previous year and targets for the next year. For each target, progress is categorized as either “on-track” or “off-track”. If the target is off-track, or has been significantly exceeded, this needs to be explained, and the implications for next year’s target and the end-of-program target must be clarified. Each program results framework also includes a “summary of changes” section, which outlines changes made to the program in the previous year due to active management and adaptation based on results achieved.

A number of examples of this approach can be drawn from the 36 field programs that submitted results frameworks in 2014, as part of the Annual Programme Review Process:

  • The Promoting Gender Equality program in Pakistan incorporated a new advocacy and communication component, based on a need identified through RBM. The new component was introduced into the program’s results framework, together with appropriate indicators and targets.
  • The Sustainable Livelihoods and Environmental Justice program in India adapted targets around the percentage of farmers using organic inputs only, based on information from monitoring done in 2013. Monitoring indicated that increasing government subsidies on agricultural inputs to increase production was creating a more difficult working environment for the program.
  • The Sustainable Livelihoods and Environmental Justice program in the Democratic Republic of Congo shifted its emphasis from improving individual production to collective processing, storing, and marketing of crops. Monitoring revealed that although producers had significantly increased their individual production capacity, they had not yet managed to increase their incomes.
  • In the Promoting Gender Equality and Addressing HIV and AIDS programs in Malawi, a new dimension was added to ensure changes would be sustained over time, following quicker than expected initial progress. Monitoring indicated that more community leaders than religious leaders were speaking out on HIV, and these groups needed separate targets to monitor their progress more closely.

NATURE AND SCALE OF CHANGE BEING ACHIEVED

Adopting RBM has also helped Trócaire to be clearer about the nature and scale of change achieved within its programs. An example is found in the annual review of the Kenya HIV program: in 2013, 4,847 people (2,367 men and 2,480 women, including 399 couples) received HIV testing and counselling (HTC) services, which was a 289 percent increase on the 2012 result. This exceeded the target of 2,000 new people receiving HTC services, including 120 couples. Because of this, the end-of-program target was revised up to 7,000 people receiving HTC services, including 600 couples.

Trócaire is also starting to aggregate results across programs more effectively. For example, land titles were secured for 80,000 households in the Democratic Republic of Congo, Guatemala, Honduras, and India in 2013. In the same year, access to water for irrigation and household consumption was provided to 64,000 households across Ethiopia, Kenya, Malawi, Nicaragua, and Honduras.

In addition, the results agenda has provided an opportunity to disaggregate by gender, to establish gender-sensitive indicators, and, if on measurement a gap in status is apparent, to set gender-equitable targets for each outcome and apply strategies to address gender-related disparities.

As Trócaire is clearer on the nature and scale of change being achieved, the organization is able to progress work on value for money and the strategic allocation of resources at program, country, and organizational levels, based on evidence of which work is achieving strong and not-so-strong results.

ATTITUDES TOWARD RBM

The organizational culture and attitude regarding RBM has changed at Trócaire. There is now a more positive attitude as staff see the benefits of RBM in designing and managing their programs, and feel much greater ownership of the RBM approach. There also appears to be greater recognition of the value of evidence in managing programs.

One reason for initial resistance toward RBM was a fear that it would lead Trócaire, as an organization, into focusing more on types of work that are easier to measure but are not necessarily types of work that Trócaire should do to bring about the change it wishes to see. Trócaire’s approach was to persist with the type work it knows is critical to bring about change, and find ways to measure this work.

Trócaire made much progress in sharpening outcomes and developing common, standardized indicators and data collection methodologies and tools for work that it had previously struggled to measure. Where appropriate, the organization found ways to quantify what would previously have been considered purely qualitative information, often by applying scalar tools. This work is led by the technical teams supporting field-based colleagues, and is informed by field-based colleagues. It is ongoing, and common outcomes, indicators, methods, and tools continue to be refined. This approach is appreciated by colleagues.

As an organization, while we have struggled to measure some kinds of work, we have not stopped work in the hard-to-measure areas, that is, we have not resorted to doing work that is easy to measure. Rather, we have continued and persisted to address the difficult issues and implement the type of work we think is necessary, and have tried to find ways to measure our progress for this work. (Trócaire program manager, Malawi)

IMPORTANCE OF RBM FOR ACQUIRING INSTITUTIONAL FUNDING

In 2013 and 2014, Trócaire submitted 35 very successful program results frameworks to Irish Aid, outlining results against targets for 2013 and setting targets for 2014. Trócaire was complimented by Irish Aid on the quality of its submissions. Compared with 2012, the turnaround time between submission of the results frameworks and disbursement by Irish Aid of its program funding for the year was relatively short.

According to Trócaire’s institutional funding coordinator, having robust RBM systems at organizational level and using RBM at program level is critical to acquiring institutional funding from major donors such as the UK Department for International Development (DFID), Comic Relief, Big Lottery Fund, and the European Commission. Increasingly, major donors want to see funding applications that are focused on results rather than outputs, and this has been the major change in recent years. In the assessment of funding applications there is now, compared to just a few years ago, much greater scrutiny of anticipated results and a focus on how realistic, achievable, and measureable these results are. There is also much greater scrutiny of an organization’s approach to RBM.

Trócaire’s investment in RBM has been important in acquiring significant funding from major donors (including several major grants from UK-based donors over 2014) and for complying with donor reporting requirements. Major donors also require that the achievement of results is clearly costed (budgeting by outcome).

Challenges

Among the challenges for Trócaire is aggregating results across programs and communicating the big picture. There are currently a wide range of outcomes and indicators even in similar bodies of work. Trócaire is starting to use more standardized outcomes and indicators across programs, and this should primarily help to improve program design and program management, but should also facilitate the aggregation of results across programs and communicating the big picture.

Further work needs to be done to embed results in the humanitarian crises response work. RBM training has been delivered to humanitarian crises preparedness and response colleagues, and results frameworks have been developed and are being used for five humanitarian crises response programs that receive Humanitarian Programme Plan funding from Irish Aid. However, more work needs to be done to develop humanitarian crises response-specific guidelines and templates. The challenge is to take the learning from this experience and adapt it to the context of humanitarian crises response work, where the sudden-onset nature of the work, as well as very different time frames, will necessitate an adapted approach.

A more systematic approach is needed to support partners in RBM, and to engage them in monitoring and evaluation. Experience has shown that investing in supporting partners to use RBM is very beneficial, both for partners and the overall program. However, this has not yet been done systematically in all countries and for all partners, resulting in different levels of understanding of and engagement in RBM. Measurement of results in Trócaire programs, particularly for ongoing monitoring, relies on all partners having the capacity to gather and report data to a consistently high standard.

While RBM has been acknowledged as an important opportunity for recognizing, measuring, and responding to gender inequalities, more needs to be done to interpret sex disaggregated data in results frameworks in order to strengthen gender mainstreaming across programs. The Trócaire/Oxfam/Design for Peace collaboration on ENABLR is an opportunity to address this. ENABLR is website-based software tool designed to support program staff to identify appropriate strategies and actions for mainstreaming gender into livelihoods, governance, and humanitarian programs. It will initially be used as part of the process for program design, and as the tool is further developed it is envisaged that it should also support the monitoring and evaluation of gender mainstreaming approaches.

As already outlined, work has begun to incorporate considerations of strategic and accountable resource allocation organizationally, and discussions on results and lessons learnt at country and program level are linked more rigorously to analysis of budget allocations. However, further work needs to be done to link the results Trócaire is achieving to the strategic allocation of resources and to ensure that results consistently inform management decisions. Trócaire needs to capitalize on the opportunity presented by having strong results information to better articulate value for money.

Conclusions

RBM has provided significant learning for staff and local partners, and is yielding positive results in terms of clearer thinking around the aims and objectives of programs, better and more nuanced design of programs, better managed programs, higher quality annual reports, and more robust evidence of impact. Moving forward, there will need to be ongoing capacity building to help staff manage programs by results and, through staff, ongoing capacity building with partners.

Fewer and more standardized program outcomes and indicators will be needed, to enhance program quality and facilitate aggregating and “global reporting.” These standardized outcomes and indicators will be linked to tried and tested data collection methodologies and tools.

Further investment in digital data collection – using technology to collect and analyze data, both in “field” programs and outreach work in Ireland – will also be needed. Trócaire’s pilot work, using digital data to monitor progress in programs, concluded that it offers substantial efficiency gains (but that technology should supplement rather than replace human processes and skills, and these should be developed before introducing digital data). Trócaire will harness the potential of technology to clearly communicate results: data visualization innovations have the potential to make results more accessible to a wider audience.

There is much speculation in the sector that some major donors will introduce payment by results (PBR). This form of financing would make disbursement of funds contingent on achievement (and independent verification) of results, with certain donors disbursing funds at agreed intervals only when set targets have been achieved. PBR is a cross-government reform priority in the UK, with the UK Department for International Development (DFID) among leaders ensuring this approach gained initial momentum internationally (DFID 2013). More recently, PBR has been heavily criticized by key stakeholders in the sector, and the likelihood of it becoming established practice for international development and humanitarian programs, and the implications for Trócaire and agencies like Trócaire, are as yet unknown. However, this reinforces the imperative for further investment in RBM, so that Trócaire will continue to consistently refine and improve articulation of its approach and desired outcomes. Trócaire will need to use the most appropriate indicators, methodologies, and tools to set targets that are as measureable and realistic as possible. The setting and achieving of accurate and realistic targets will become ever-more important.

While there has been a strong focus on RBM for Trócaire’s international development work and development education work, investment in RBM has been ongoing in other aspects of public engagement. Further work will be done to build on the RBM training conducted with colleagues in humanitarian crises response work. There has also been considerable interest from staff and managers across other parts of the organization to strengthen and embed the approach. Ultimately, there is and will continue to be sustained investment in improved approaches to monitoring and evaluation, to ensure that the commitment to RBM remains an achievable and effective method of managing programs and projects, and does not overwhelm program managers or partners, or distract from program or project implementation.

About the authors

Eoghan McKernan is EU Programme Manager at the Irish Centre for Fetal and Neonatal Translational Research (INFANT), University College Cork and also leads INFANT’s Global Health Programme on collaborative perinatal research in Tanzania. He has spent over 12 years in international development, including the UN and Trócaire where he was Strategy and Impact Adviser (2009 - 2014) and also has significant experience in research and clinical research support. Eoghan has a PhD in Food Science and Technology from UCC and is PRINCE2 certified.

Karen Kennedy is the Head of Strategy and Impact at Trócaire, having previously worked as an advisor within the Strategy and Impact Unit. She has ten years’ experience in international development and humanitarian response, working for agencies such as Médecins Sans Frontieres and International Rescue Committee. She holds a PhD in Electrical Engineering and has worked with Eirgrid and Siemens Power Transmission and Distribution. She is PRINCE2 certified.

Allison Aldred is currently CEO of Suas Educational Development. Prior to this role, she was Head of Strategy and Impact at Trocaire (2009 - 2014). Before joining Trocaire, Allison spent nearly ten years with Oxfam GB including seven years as Regional Director for South Asia. During this period, she also acted as Oxfam’s global lead for Gender Equality. Allison is currently Chairperson of the West Cork Women Against Violence Project. She has earned a degree in Political Theory and also a Master of Business Administration from London Business School.

DECLARATION OF CONFLICTING INTEREST The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article. FUNDING The author(s) received no financial support for the research, authorship, and/or publication of this article.

References

Serrat, O. & Buentjen, C. 2015, ‘Ramping up results-based management in the Philippines’, Knowledge Showcases, June 2015, Issue 64, Asian Development Bank.

Bester, A. 2012, Results based management in the United Nations development system: progress and challenges, report prepared for the United Nations Department of Economic and Social Affairs for the Quadrennial Comprehensive Policy Review, UNDESA, New York.

Binnendijk, A. 2000, Results based management in the development co-operation agencies: a review of experience, background report, Development Assistance Committee Working Party on Aid Evaluation.

Center for Theory of Change 2013, TOC origins, viewed 10 September 2014, http://www.theoryofchange.org/what-is-theory-of-change/toc-background/toc-origins/

Danida 2014, Results based management, viewed 10 September 2014, http://um.dk/en/danida-en/results/results-based-management/

Department for International Development 2013, ‘Sharpening incentives to perform: DFID’s strategy for payment by results’, policy paper, DFID, London and Glasgow.

Department of Foreign Affairs, Trade and Development Canada 2011, Results-based management policy statement 2008, viewed 9 September 2014, http://www.international.gc.ca/development-developpement/partners-partenaires/bt-oa/rbm08-gar08.aspx?lang=eng

Department of Foreign Affairs, Trade and Development Canada 2014, Results-based management, viewed 8 September 2014, http://www.international.gc.ca/development-developpement/partners-partenaires/bt-oa/rbm-gar.aspx?lang=en

Dóchas 2014, Results Learning Group, viewed 22 August 2014, http://www.dochas.ie/our-work/learning-groups/results-learning-group

Drucker, F.P. 1954, The practice of management, Harper & Row, New York, NY.

Eyben, R., Guijt, I., Roche, C. & Shutt, C. 2015, The politics of evidence and results in international development: playing the game to change the rules, Practical Action Publishing, Rugby. http://dx.doi.org/10.3362/9781780448855

Hatton, M.J. & Schroeder, K. 2007, ‘Results-based management: friend or foe?’, Development in Practice, vol. 17, no. 3, pp. 426–32. http://dx.doi.org/10.1080/09614520701337160

Hulme, D. 2010, ‘Lessons from the making of the MDGs: human development meets results-based management in an unfair world’, IDS Bulletin, vol. 41, no. 1, pp. 15–25. http://dx.doi.org/10.1111/j.1759-5436.2010.00099.x

Ika, L.A. 2012, ‘Project management for development in Africa: why projects Are failing and what can be done about it’, Project Management Journal, vol. 43, no. 4, pp. 27–41. http://dx.doi.org/10.1002/pmj.21281

Irish Aid 2013, ‘Irish Aid Programme Grant 2012–2015 Programme Cycle Management Guidelines’, updated December 2013.

Meier, W. 2003, ‘Results based management: towards a common understanding among development cooperation agencies’, discussion paper ver. 5.0, prepared for the Canadian International Development Agency, Performance Review Branch, for consideration by the DAC Working Party on Aid Effectiveness and Harmonisation.

OECD 2014, Paris Declaration and Accra Agenda for Action, viewed 12 August 2014, http://www.oecd.org/dac/effectiveness/parisdeclarationandaccraagendaforaction.htm

Open Forum for CSO Development Effectiveness 2014, Istanbul Principles, viewed 4 September 2014, http://cso-effectiveness.org/istanbul-principles,067

UNESCO 2011, ‘Results-based programming, management and monitoring (RBM) approach as applied at UNESCO: guiding principles’, UNESCO, Paris.

USAID 1999, Results-oriented assistance: a USAID sourcebook, viewed 15 August 2014 http://pdf.usaid.gov/pdf_docs/PDABR702.pdf

Vahamaki, M., Schmidt, M. & Molander, J. 2011, Review: results based management in development cooperation, Riksbanken Jubileumsfond, Stockholm.

Refbacks

  • There are currently no refbacks.