User Tools

Site Tools


mel_approach_principles_and_standards

This is an old revision of the document!



CARE's global approach to Monitoring, Evaluation, Accountability and Learning

CARE's global approach to Monitoring, Evaluation, Accountability and Learning

MEL Approach, Principles and Standards - English
Last updated (July 2019)
2019_july8_mel_approach_principles_and_standards.pdf
MEL Approach, Principles and Standards - Spanish
Last updated (October 2017)
care_2017_october_26_mel_approach_principles_and_standards_spanish.docx

This document outlines the approach, principles and operational standards that guide CARE’s Monitoring, Evaluation, Accountability and Learning practices for projects and initiatives implemented around the world, which can also be applied to the work CARE does with and/or through partners. It combines key elements from longstanding policy documents like the CI Program Principles and Program Standards,the CARE Evaluation Policy, and others; together with elements from the broader MEL debate in development and humanitarian work. As a result, it provides with an updated view on how CARE defines and operationalizes MEAL.


Summary


THE FOUNDATIONS OF CARE’S MEAL APPROACH (October 2017)

The Foundations of CARE´s MEAL approach

The foundation of CARE’s approach to Monitoring, Evaluation, Accountability and Learning is the recognition that we work in very dynamic and complex contexts, where lasting social change does not follow a linear timeline or a single pathway, where multiple stakeholders interact and influence each other as well as our interventions, and where there are constant adjustments in social, economic, structural, environmental or other dimensions that we must be critically aware of and adapt to (see figure).

{{  :3083fe5ae464e6c930fd1d2c405ac19b.png?273x198 Under these circumstances, our organizational ability to demonstrate the impact of our work and explain how we contribute to lasting change lays in the ability of CARE’s projects and initiatives to put dynamic MEAL systems and practices in place. That means, MEAL systems that continuously generate comprehensive explanations and evidence on the way we think about a situation or problem and its underlying causes; a process of desired social change; how CARE’s interventions contribute to that change and how other factors and critical preconditions take place in society in order for that change to happen.

Social change understood as the overcoming of poverty, enjoying equitable opportunities for women and men, being part of inclusive development processes and being able to continuously transform in response to new hazards and opportunities.

In summary, MEAL systems and practices become critical to “unpack” the WHO, WHAT, HOW and WHY of social change:

WHO are the specific populations (women, girls, men and boys) ultimately experiencing lasting change, and who are the other actors facilitating that change?

WHAT changes are those populations experiencing?

HOW and WHY are those changes happening and what role does CARE and other actors play in facilitating those changes?

Applying this approach also implies that CARE’s MEAL systems and practices put special emphasis on explaining social change and impact as a combination of our actions plus the influence of other critical factors that make a change process possible (contribution), and only when appropriate, our MEAL systems focus on purely explaining social change as fully attributed to CARE’s actions (attribution). Although explaining attribution can often be considered a more robust way to show evidence of impact influenced by a set of interventions, we strongly believe that CARE’s contribution to social change is influenced and enriched by multiple actors and contributing factors. Therefore, our potential to explain complex change and multiply impact is enriched when we are able to complement a rigorous causal analysis with the explanation of the many elements influencing change, and the role of different actors have in facilitating that change.

Important Note:CARE’s MEAL approach includes the following definitions:

  • Long-term or ultimate outcomes - Impact: includes sustainable, significant and measurable changes in well-being, materialized in lasting changes on poverty and social injustice conditions of a particular population. Changes at the impact level are influenced by those factors directly addressed by a project or initiative, as well as other factors.
  • Immediate and intermediate outcomes - Outcome: includes changes on individual behaviours (e.g. individuals putting into practice new knowledge, attitudes or commitments) and changes that are structural or systemic (e.g. policy changes, new practices in service provision), that can be seen in different populations. Outcomes are often a result of what participants do on their own, influenced by the actions of a project or initiative or other factors.
  • Output: includes the direct results of activities implemented by a project or initiative. Outputs may refer to: a) The results of training, such as the number of women trained in improved nutrition, farmers in improved agricultural techniques, etc. b) Capacity building, such as the number of extension staff trained, water systems built, committees established, etc.; c) Service outputs, such as an increase in the number of program locations; d) Service utilization, such as the number of people fed, or number or patients treated. Outputs are the products a project or initiative generates through the implementation of its activities.
  • Inputs: Includes the set of resources that are needed by a project or initiative in order to deliver its commitments. These include the human and financial resources, physical facilities, equipment, materials, logistics, in-kind contributions and operational polices that enable services to be delivered.

In CARE, we acknowledge the fact that different actors use different definitions, nevertheless, note that this doesn’t affect the way the MEAL approach, principles and standards are defined and can be applied.


MEAL PRINCIPLES

MEL PRINCIPLES

The fundamental propositions behind our Monitoring, Evaluation and Learning systems and practices in CARE include:

MEAL systems and practices should be conducive to Accountability, by generating solid and accessible evidence that clearly and transparently explains CARE’s work, the reach of its actions (what we do, where we work and the people we reach) and CARE’s impact story (our contribution to impact and outcomes); and by deliberately setting up mechanisms to engage and involve multiple actors and incorporating the perspectives of women, men, girls and boys in decisions and actions throughout the life of a project or initiative. This includes accountability to participants, donors and many other stakeholders.

MEAL systems and practices should be conducive to Learning and potentially to Multiplying Impact, by generating and documenting evidence that strengthens the organizational memory and expertise, plus energizes learning dialogues and the identification of successful models and/or opportunities for scale-up.

MEAL systems and practices should be conducive to Adaptation, by tracking, interpreting and summarizing key data related to changes in social, economic, structural, environmental or other dimensions that a project or initiative should be critically aware of and constantly adapt to.

MEAL systems and practices should always balance purpose, methodological rigor and capacity, by identifying the most appropriate combination of methods to address: purpose (contribution / attribution), evidence needs and uses, resources, capacity, technology requirements and other factors.

IMPORTANT:In the current MEAL debate, we may find inclination towards assessing attribution, sometimes without sufficient resources or without considering the sensibility of applying certain methods in certain context. Being clear on the methodological appropriateness helps solving challenges around this. For example, if an intervention is seeking to generate evidence of impact, plus also validate a model or innovation, the selection of evaluation methods will be influenced by CARE’s global program priorities, the nature of the intervention, the rigor required, donors requirements, uses of the evidence, the capacities in place and resources available, etc. Normally, this results in the combination of quantitative and qualitative methods.

MEAL systems and practices should always consider ethical implications and be conducive to gender equality, by ensuring honesty, consent and integrity of all MEAL practices and MEAL methods selected; always respecting the security and dignity of the stakeholders with whom CARE works; incorporating gender and power elements when monitoring and evaluating; generating evidence disaggregated by sex, age and other relevant diversity categories, etc.

MEAL systems and practices should be dynamic and lead to action, by intentionally planning and executing various iterations of monitoring, evaluation, accountability and learning moments throughout the life of a project or initiative, and stablish a clear connection with informed decision-making spaces, and actions.

MEAL systems and practices in projects and initiatives should contribute to CARE’s global evidencing efforts, by generating evidence and engaging on reflection around CARE’s collective reach and impact story, globally.


MEAL STANDARDS - SUMMARY LIST

MEAL STANDARDS

From a practical perspective, the definition of a solid MEAL system for projects or initiatives in CARE should include the application of the following MEAL standards:

Design your MEAL system based on a clear theory of change and evidence needs.

Have a clear definition of participants and the mechanisms to register/count/track/report participants’ data

Define a meaningful and manageable set of quantitative and qualitative indicators and/or questions for impact, outcomes and outputs in each participant group, and the methods to track them

Define the monitoring and evaluation moments and methods that best ensure robust and comparable tracking of outputs, outcomes and impact

Ensure your evidence can be translated into learning and support on the identification of potential for scale

Make your evidence accessible, and ensure your MEAL practices are participative and responsive to feedback.

Use your MEAL system to continuously read the context and adapt to it.


MEAL STANDARDS - EXTENDED EXPLANATIONS AND GUIDANCE FOR THEIR APPLICATION

MEAL Standards

Design your MEAL system based on a clear theory of change and evidence needs.

Projects and initiatives are normally designed based on a holistic analysis of context and stakeholders, plus a theory of change or any similar type of comprehensive explanation of the desired changes, the different pathways to get to the desired change and causality. The core of your MEAL system should be designed to continuously test the Theory of Change of the project and initiative, being able to answer questions like the following:

  • What are the key outputs and activities the MEAL system will track in order to inform if the implementation of activities is in the right track, and reaching the expected participants? (direct and indirect participants)
  • What are the key qualitative and quantitative changes (impact and outcomes) the MEAL system will track in order to inform if CARE is contributing to significant and lasting changes? Which pathways of change and causality relationships will we track? Who are the actors we will focus on when tracking those pathways? (impact and target populations)
  • What are the key risks and assumptions the MEAL system will track and review during implementation in order to ensure the project or initiative is responsive to the context? How will unintended consequences or emerging changes be part of the continuous testing of the theory of change?
  • What are gender, governance and resilience considerations the MEAL system will track?

Have a clear definition of participants and the mechanisms to register/count/track/report participants’ data

Projects or initiatives normally have well defined impact group(s) - those individuals that will ultimately experience impact or lasting change and target group(s) - those individuals whose behaviors or actions will influence the realization of changes for the impact groups (see figure in next page). If your project or initiative has a theory of change or has done a stakeholder mapping, the identification of impact and target groups would normally come from there.

Important note here: When looking at impact and target groups in your project/initiative, please remember that they must be identifiable as individuals that can be described and counted. They cannot be identified in general terms like households, families, groups, communities, organizations or other.

Here some examples of questions that can help you confirm if you are looking at concrete individuals:

* For the impact group: is the impact group composed of only some individuals from a household? (e.g. mothers and children under 5); or the entire household? (e.g. all individual members of food insecure families); or a clear portion of the population in a given location? (e.g. only women and girls of school age in the area where the project intervenes).

* For the target groups: who are the individuals that compose the target groups? Is it a clearly defined group of community/religious leaders? Or a specific number of decision makers or government officials? Or a number of members of civil society organizations? Or a clear portion of the population in a community? Or all the population in a given location?

Once the project or initiative has clarity on who the impact groups and target groups are, the next step would be for the project or initiative to define how it will count/track and report on these participants. In CARE’s language, this is where the concept of participants REACHED and participants IMPACTED comes into play:

Participants REACHEDParticipants IMPACTED
* Refers to all those individuals that a project/initiative connects with as it implements its activities and delivers outputs.

* Participants REACHED may include:

1) individuals who are directly involved in activities implemented by the project or initiative, receiving support, services, goods, resources or other, from CARE or partners
2) individuals who are not directly involved in activities implemented by the project or initiative, but still indirectly connect with the outputs resulting from those activities.

* Participants REACHED can be classified as DIRECT or INDIRECT participants, however, this classification greatly depends on the modality of implementation that the project/initiative adopts, and other factors.

* For detailed guidance on how to define participants REACHED and IMPACTED for different modalities of implementation, please refer to this guidance noteguidance note
* Refers to all those individuals who, as a result of the materialization of the goals of a project or initiative, experience lasting change (impact or outcomes).

* Depending on the impact or outcome metrics/indicators your project or initiative uses to measure lasting change, the participants IMPACTED could include:

1) Individuals from the impact group, experiencing lasting change in their lives (e.g. households graduating from extreme poverty; families becoming food secure; children under 5 no longer stunted; women generating income and accessing education; women participating in joint decision-making in the household, etc.)
2) Individuals from the target groups whose changed behaviors are also part of lasting change (e.g. local leaders/men and boys rejecting intimate partner violence).

* For detailed guidance on how to define participants REACHED and IMPACTED for different modalities of implementation, please refer to this guidance noteguidance note

Note that, within the definition of participants REACHED, we are just looking at tracking/counting and reporting on individuals that are involved in the activities of a project or initiative. We are not counting/tracking/reporting if/how these individuals are experiencing important changes in their lives, like impact/outcomes (e.g. a participant that is receiving training would be counted as REACHED and, unless there is an evaluation process to determine if the knowledge acquired in the training has led to an outcome or impact, this individual would not yet be counted as participant IMPACTED).

The tracking of participants impacted, normally needs to be supported by an evaluation process (external or coming from a solid monitoring of outcomes) and requires measurement of outcome or impact indicators that the project or initiative has defined since its design (see next standard).

Define a meaningful and manageable set of quantitative and qualitative indicators and/or questions for impact, outcomes and outputs in each participant group, and the methods to track them.

Make sure to incorporate at least one of the CARE’s Global Impact and Outcome Indicators, together with any other supplementary indicator that is relevant or required (e.g. by the donor). Recommendation: Try to avoid creating new indicators or indexes.

Quantitative indicators or questions would regularly help you demonstrate WHO are the specific populations experiencing change? (e.g. women of reproductive age; policy makers) And WHAT changes are they experiencing? (e.g. increased safe births; improvements in policy that guarantees women’s access to quality SRMH services). Qualitative indicators or questions would regularly help you demonstrate HOW and WHY are those changes happening? What role does CARE and other actors play in contributing to those changes? (e.g. is the increase in safe births explained by CARE’s strategy and work with health centers and communities? Or is it the product of changes in how decision makers recognize the importance of women accessing quality SRMH services ant take action on their own?).

Based on the indicators and questions you select for tracking impact, outcomes and outputs, define the most appropriate combination of methodological approaches to track them.

Cost-effectiveness: If the project or initiative has then intention to track cost-effectiveness, this is where you will need to design a tracked record of expenditures utilization linked with the identified outcome indicators, in order to be able to demonstrate the cost-effectiveness of certain strategies or outcome areas.

Define the monitoring and evaluation moments and methods that best ensure robust and comparable tracking of outputs, outcomes and impact.

4.1 Monitoring outputs and participants

Define the moments, tools and resources used throughout the life of the project or initiative to track outputs from all those key activities being implemented (e.g. health staff from health services participating in training). While collecting and analyzing data at this level, the MEAL system won’t generate explanations related to impact or outcomes but will regularly ask if all the activities and outputs are the most appropriate and if they are really setting the bases towards the expected outcomes and impacts.

Important considerations when monitoring participants:

* Participants are always individuals. Even if our projects or initiatives work with households, communities or institutions, these are always composed of individuals, therefore, should ultimately monitored as individuals. * One individual can be reached by one or more project or initiative in one particular context. The monitoring actions should be aware of duplications with other projects or initiatives, and establish the mechanism to report data without double counting. * Participants’ data should normally be disaggregated by sex, age and potentially by disability or any key criteria related to the problem or vulnerability the project or initiative seeks to address. Estimations based on statistical references (e.g. census) are not always the most accurate measure. If the disaggregation is made using estimates, the source of the ratio must be explained. * In projects or initiative implemented in the course of multiple years, the total participants in a particular year should be cumulative and single counted (existing and new participants). Even though it is important to know the incremental process, participant’s information is not normally aggregated year by year.

4.2 Monitoring of Outcomes

Define the moments, tools and resources used throughout the life of the project or initiative, to track key behavioral changes in some actors or strategic elements that set the causal linkage between outputs and outcomes and impact. Outcome monitoring helps generating indicative information (qualitative and quantitative) of what’s changing and what’s not / what’s working and what’s not, as the project or initiative advances towards the expected outcomes. For example, what happens after health staff participates in training? Do their behaviors change? How does changes in behavior favor women’s access to SRMH services? Outcome monitoring can be a continuous action (e.g. performing participant observation or doing informal interviews constantly), or a periodic action (e.g. applying an annual questionnaire or survey). In all cases, outcome monitoring may or may not have the same levels of representativeness of an evaluation, nevertheless, it does provide with important indications of progress and learning around the way the project is progressing towards contribution to change, the appropriateness of the strategies used and the validity of the assumptions in the theory of change.

Important considerations when operationalizing outcome monitoring actions:

  • Is the volume of data, the frequency with which data is collected and the moments in time monitoring actions are undertaken, and the most useful for the project or initiative?
  • Are the monitoring actions (collecting, reporting or analyzing data) considering the availability of time and predisposition of project staff or to project participants?
  • Will all the data generated by monitoring actions be used and disseminated, and will inform decisions on the implementation or the theory of change of the project or initiative? Note: If that is not the case, you may be collecting more data than you actually need.
Value for Money: If your MEAL system needs to look at “Value for Money”, the monitoring component should incorporate elements that look at other elements like economy (quality of inputs), efficiency (delivery of outputs), effectiveness (extent to which outputs are converted into outcomes and impacts) and equity (extent to which most vulnerable groups are reached) of the actions implemented by the project or the initiative.

4.3 Evaluation

Define the moments, tools and resources used throughout the life of the project or initiative, to objectively assess its relevance and fulfilment of objectives, its efficiency, effectiveness, impact and sustainability, and/or its worth or significance (based on the OECD/DAC definitions). Evaluations in CARE projects and initiatives can be carried out for different purposes and take a variety of forms (see descriptions below). Nonetheless, all evaluations need to provide with substantiated evidence of the changes that took place as a result of a project or initiative’s actions, and a plausible explanation of how CARE’s actions contributed to the materialization of those changes.

  • Formative evaluations: carried out during implementation of a project or initiative, intended to improve a project´s performance, informing necessary adjustments of project in relation to project design, planning, resources, approaches and methodologies, and capturing lessons and promising practices that inform decision-making (e.g. real time/mid-term evaluations of any project or initiative).
  • Summative or End-line evaluation:often carried out at the end of a project, intended to assess the extent to which expected outcomes have materialized and assessing its significance or relevance (end-line evaluations).
  • Impact evaluations: carried out either during or after the implementation of a project or initiative, intended to demonstrate impact in a cause-and-effect manner to an intervention. In impact evaluations, the focus shifts away from what CARE is doing, to observe and track the changes that take place in the lives of the impact groups, and how these changes come about. Impact evaluation normally entails a step further than any other type of evaluation and implies a deeper look to the participants and the changes they experience, plus collaborating with others in order to explain how these changes were facilitated by the project or initiative. As a result, it directs all is attention to test the theory of change behind the project or initiative and demonstrate how CARE contributes to that.

    Important considerations when operationalizing evaluations:
  • Evaluations should provide with complete and comparable assessments of the before-after or with-without situation.
  • Evaluations should assess desired as well as unexpected outcomes.
  • Evaluations can be conducted or supported by qualified professionals who establish and maintain credibility in the evaluation context. However, CARE staff should be highly involved in the whole evaluative process from the very beginning, not only to guarantee ownership of the process but also to open opportunity to strengthen MEL capacities and to learn.
  • Evaluation results need to be processed and reported in multiple ways and addressing different stakeholder needs and purposes. Evaluation results should be accessible for learning and for encouraging the project and participants to rediscover, reinterpret, or revise their understandings, plans and behaviors.

Ensure your evidence can be translated into learning and support on the identification of potential for scale.

Monitoring and Evaluation actions normally generate a great amount of data and evidence, therefore, can naturally contribute to a structured body of information and knowledge inside and outside of. However, data and evidence can only be useful for learning and for multiplying impact, when data and evidence is adequately organized, processed, analyzed, discussed and shared.

Important considerations to link monitoring and evaluation with learning:

  • Define a learning agenda from the very beginning of the project or initiative, around the following question:
    • What is it that we want to learn from the implementation of this project or initiative?
    • Will the data or evidence to be captured by the MEL system support learning in general or advance critical learning on a particular issue?
    • Will it potentially generate evidence for multiplying impact?
    • Is the monitoring and evaluation data sufficient and relevant enough for that learning or will we need additional research in a particular area?

Note that we can’t learn every single aspect of our work. Prioritization in a learning agenda is critical.

  • Make sure your monitoring end evaluation data and evidence is well organized and hosted in a safe and accessible system or platform.
  • Open specific moments in the life of the project or initiative, to share and discuss findings in ways that are understandable and useful to various stakeholders - participants and partners, staff of various units within the CARE consortium, as well as donors.
  • Whenever possible, include external actors on monitoring or evaluation teams (e.g. project staff, representatives of other CARE projects or partner agencies, etc.).

Make your evidence accessible, and ensure your MEAL practices are participative and responsive to feedback.

CARE’s commitment to accountability implies that projects and initiatives promote transparency in their actions, information and decisions, encourage participation from different stakeholders to shape their work, and deliberately open channels for feedback and take action based on feedback.

Important considerations when linking monitoring and evaluation with accountability:

  • Ensure your MEAL actions balance the moments for data/evidence collection with moments for actors to provide feedback to CARE, and make sure to connect this feedback to the appropriate instances, so that feedback is always followed by action.
  • Define how and in which moments will MEAL staff, program managers and other CARE and non-CARE actors will engage and collaborate in all the different steps of generating and using data, analyzing and responding to feedback, as well as making decisions for adaptive management.
  • Ensure the MEAL system embeds a feedback and complaints mechanism that is comprehensive and in line with global standards (e.g. Core Humanitarian Standards).
  • Make sure the targeting strategy of the project or initiative and the definition of participants promote equity and address the needs of the most vulnerable groups.
  • Make sure key information generated by your MEAL system is accurately reported and available for different audiences (examples: PIIRS data, the Reach and Impact Map, etc.)
  • Make sure your evaluations are well documented and publicly available in CARE’s Electronic Evaluation Library http://www.careevaluations.org.

Use your MEAL system to continuously read the context and adapt to it.

Adaptive approaches are increasingly and undeniable relevant to address complexity in the contexts in which we implement projects and initiatives. Our capacity to adapt covers many other areas of organizational culture, structures, processes and capacities that go beyond MEAL purely. However, MEAL systems can be highly instrumental for adaptation.

Important elements to consider when linking MEAL to adaptive management:
* Your MEAL practices need to be agile and have the capacity to collect data, generate evidence, identify changes and generate recommendations more frequently. * The MEAL system should include regular review points when monitoring and feedback data is assessed against the theory of change, so that adaptation can occur accordingly. * Your MEAL system should dedicate considerable effort to rapid learning and very agile feedback, in order to inform changes. * Your MEAL system needs to be flexible, adjusting indicators, methods, tools and resources based on potential changes of the overall design of the project or the initiative. * Your MEAL system should be clearly linked with decision making instances, in order to make sure that data and evidence signaling need for adjustments are taken into action.

mel_approach_principles_and_standards.1563274087.txt.gz · Last modified: 2019/07/16 13:48 by admin