Terms of Reference Final Evaluation of USAID-BHA

RAPID (Responding to Pakistan Internally Displaced) Fund - III

[Agreement No: Award 720FDA18CA000050]

January 2024

Summary

Concern is commissioning the performance evaluation of RAPID III covering the period from October 2018 to August 2024 to provide a holistic understanding of its achievements, challenges, and overall impact in relief and early recovery responses and building capacities of the organizations and DMAs for effective humanitarian response. This performance evaluation aims to thoroughly examine the outcomes and impacts of the relief activities and capacity-building initiatives supported by RAPID Fund III. Through applying OECD DAC criteria and comparative analysis with similar funding mechanisms in Pakistan, the evaluation will assess the relevance, appropriateness, coherence, effectiveness, efficiency, partnership, and overall impact of RAPID Fund III in addressing the relief and early recovery needs of disaster-affected populations. The consultants or firms will employ quantitative and qualitative methodologies to ensure a comprehensive evaluation and draw evidence-based conclusions.

The evaluation process will generate several valuable outputs, including inception and evaluation reports. The inception report will outline the evaluation's methodology, including the data collection techniques and analytical framework. On the other hand, the evaluation report will present a comprehensive analysis of the findings, highlighting the performance of and lessons learned from the implementation of RAPID III. These outputs will contribute to ongoing efforts aimed at enhancing the humanitarian response in Pakistan. The findings and recommendations derived from the evaluation will assist Concern, USAID/BHA, and other relevant stakeholders in refining and strengthening future humanitarian interventions.

Background:

Concern Worldwide is an international non-governmental organization (INGO) committed to alleviating suffering and striving for the eradication of extreme poverty in the most impoverished countries across the globe. Since 2001, Concern has been actively engaged in Pakistan, implementing programs encompassing emergency response, resilience building, and long-term development initiatives throughout the country.

With the generous support of the United States Agency for International Development's Bureau of Humanitarian Assistance (USAID-BHA), Concern is implementing the Responding to Pakistan's Internally Displaced (RAPID) Fund phase III (RAPID III). It is a flexible, readily available, easily accessible, and expeditious funding mechanism to award sub-grants to local/national and international NGOs for effective humanitarian response in Pakistan. The collaborative efforts, strategic partnerships, and quick implementation have effectively contributed to improving the overall protection and well-being of disaster-affected people in Pakistan.

RAPID III also strategically allocates resources towards enhancing the capacities of local/national organizations and Disaster Management Authorities (DMAs) in Pakistan, promoting robust and effective humanitarian response (EHR). By imparting individuals and institutions with essential competencies and expertise, RAPID III endeavors to cultivate a culture of professionalism and maintain a trajectory of constant advancement in humanitarian operations^[1]. Hence, RAPID III has two primary objectives:

Objective I: Through small-scale, short-term grants, support local and international non-government organizations (NGOs) responding to emergency needs throughout Pakistan.

Objective II: Strengthen the technical, management, and programmatic skills of local non-governmental organizations, as well as local disaster management authorities, to increase the quality of their internal operations and their standards of work on the ground.

Under objective I, RAPID III responded to several emergencies between October 2018 and June 2023, and made a positive impact on the lives of over 2 million disasters-affected people in Pakistan. A synopsis is provided below,

▪       In response to the severe drought in Sindh and Balochistan (2019 – 20), RAPID awarded 13 projects, consisting of seven projects in Sindh and six in Balochistan. These projects were successfully implemented in partnership with 13 local organizations, targeting three districts in Sindh (Tharparkar, Sanghar, and Umerkot) and two districts in Balochistan (Kallat and Chaghi). Collectively, the drought response provided life-saving assistance to 174,708 vulnerable people under Multi-Purpose Cash Assistance (MPCA), agriculture and food security measures, nutrition support, and Water, Sanitation, and Hygiene (WASH) sectors across both provinces.

▪       After approval from BHA, RAPID III swiftly initiated relief activities for COVID-19 pandemic (2020 – 21) in 11 districts, comprising two districts in Sindh (Hyderabad and Sanghar), three in Balochistan (Chaghi, Quetta, Killa Abdullah), and six in Khyber Pakhtunkhwa (Bannu, Hangu, Kohat, Peshawar, Mardan, and Swat). Collaborating with six local partners, RAPID's COVID-19 response made a significant impact on the lives of 963,618 individuals through health, WASH, S&S, and Multi-Purpose Cash Assistance (MPCA).

▪       Recently, in response to the 2022 flood crisis, RAPID III, after BHA’s approval, took proactive measures by awarding 23 projects, of which 12 projects were implemented in Sindh, 10 in Balochistan and one across the country in Protection Sector. These projects were implemented in collaboration with 22 local, national, and international partners, covering a total of 16 districts, including nine in Sindh and seven in Balochistan (2022 flood response). By addressing critical needs in the Water, Sanitation, and Hygiene (WASH), Multi-Purpose Cash Assistance (MPCA), and Shelter and Settlement (S&S) sectors, these projects successfully benefited 748,654 individuals in both provinces.

Within the framework of objective II, RAPID III has successfully enhanced the capacities of 977 individuals in Sindh and Balochistan, consisting of 481 government officials and 496 NGO staff, in various critical areas like Effective Humanitarian Response, Quality Assurance and Accountability to Affected Populations (Q&AAP), and operational capacity strengthening. Other domains of these capacity-building efforts include MIRA (Multi-Sector Initial Rapid Assessment), SIMEX (Simulation Exercise), and Urban Resilience Planning.

In addition to providing trainings, RAPID III has diligently worked towards bolstering the institutional capacities of these entities. This has been achieved through comprehensive reviews, refinements, and the development of guidelines, policies, and procedures in essential areas such as Monitoring and Evaluation (M&E), Financial Management, Chart of Accounts, Procurement, Human Resources (HR), and Code of Conduct (CoC), etc. These measures have been instrumental in enhancing the overall efficiency and effectiveness of the involved stakeholders.

Moreover, it is worth noting that similar trainings were also completed for 28 NGOs in four batches in Khyber Pakhtunkhwa (KP). These initiatives signify our continued commitment to strengthening capacities and promoting excellence in humanitarian response in Pakistan.

Purpose, Objectives, Scope, and Use of the Evaluation

Purpose

▪       This performance evaluation aims to enhance accountability and foster learning within the RAPID III program. Its purpose is to identify successful approaches and challenges and to provide actionable recommendations for strengthening the program. By extracting valuable lessons, the evaluation will support timely, coordinated, and principled humanitarian responses, promoting transparency and accountability among all stakeholders, thus ensuring a more effective and impactful humanitarian intervention.

▪       The evaluation will also analyze the coherence and integration between the humanitarian response and capacity-building components, assessing how they have mutually reinforced and complemented each other. It will evaluate the efficiency and cost-effectiveness of RAPID III's interventions and capacity-building efforts, considering the allocation and utilization of resources. Furthermore, the evaluation will identify lessons learned, best practices, and areas for improvement, providing recommendations to enhance program outcomes, sustainability, and future programming in similar contexts.

Objectives

▪       The evaluation will assess the RAPID III strengths, weaknesses, and areas where improvements can make a positive difference to the functioning and impact of funds. The evaluation will assess the performance of RAPID III and include concrete recommendations to make the funds fit for the future.

▪       Assess the extent to which RAPID III contributed to the provision of timely, coordinated, principled assistance to save lives, alleviate suffering and maintain human dignity. The evaluation will seek to identify the impact, results and outcomes of RAPID III in supporting a timely, coordinated and principled humanitarian response for people affected by disasters. This will include an assessment of the number of people reached by RAPID III-supported humanitarian action and the extent to which the assistance received made a difference in their lives, in particular in the areas visited.

▪       Assess the effectiveness of the capacity-building program in enhancing the preparedness and response capabilities of local organizations and disaster management authorities in the targeted districts.

▪       Evaluate the extent to which the capacity-building program has improved coordination, communication, and collaboration among local organizations and authorities involved in humanitarian response.

▪       Identify strengths, weaknesses, and areas for improvement in the capacity-building program, including recommendations for enhancing its overall impact and sustainability in building resilient local response systems.

Scope

▪       The evaluation will cover the period from October 2018 through August 2024.

▪       The evaluation will cover all four provinces (KP, Punjab, Sindh & Balochistan). Rapid will provide a detailed list of projects and their respective geographic areas to the selected firm. The firm will use appropriate methodology to select a representative samples for evaluation purpose

▪       It will assess both the objectives of RAPID III as outlined in the "Background" section of these Terms of Reference (ToRs), taking into account all humanitarian responses and capacity-building initiatives implemented during this period.

▪       While the evaluation will not extensively examine the connections between RAPID III and development programming, it will identify any noteworthy examples of best practices in this area, where they exist.

▪       This evaluation will be a performance evaluation, using both qualitative and quantitative methods to evaluate the Award against a set of criteria and questions, outlined below, but without a defined counterfactual, as may be specified in an Impact Evaluation.

Concern insists on a principle of independence in the evaluation of its programmes. The Rapid III evaluation will be conducted in an independent manner and carried out by persons without any conflict of interest in relation to the evaluation of the Award. The evaluation process must be transparent and demonstrably free from interference on the part of Concern or other external stakeholders. While Concern commits to providing all necessary facilitation to the consultant to ensure the successful implementation of the evaluation, neither the evaluators nor their reports should be subject to the control or interference by Concern staff.

Intended Users

The intended users of this evaluation are USAID/BHA, Concern Worldwide, sub-grantees (RAPID III partner organizations) and representatives from the affected population, and those who support Concern to examine its strategies, results, and overall effectiveness of its humanitarian commitments.

Evaluation Criteria and Questions

This performance evaluation will assess these issues under the following selected and agreed standard evaluation criteria:

Note: The final report/finding/synthesis should include an appraisal of how well the RAPID III has fared against each of the DAC criteria using the following grading scale, where:

Scale	Remarks
5	Outstanding Performance
4	Very good overall performance with few shortcomings
3	Good overall performance but with some minor shortcomings
2	Generally acceptable performance but with some major shortcomings
1	Barely acceptable performance with many major shortcomings
0	Totally unacceptable performance or insufficient data to make an assessment

Evaluation Methodology:

The consultants will propose a detailed methodology at the inception stages of the evaluation, including data collection, cleaning, analysis, and overall evaluation and reporting plan. The evaluators will employ a mixed methods approach, incorporating qualitative, quantitative, and participatory techniques, along with various data collection tools. Information will be derived from both primary and secondary sources, including a desk review of relevant documents, analysis of datasets, key informant interviews, focus groups, and a survey. To ensure robustness, all gathered information will be triangulated for validation purposes.

For the quantitative data collection, Concern expects the consultant to propose a probabilistic sampling approach whereby sample frames are identified according to the evaluation objectives, and a sampling strategy is proposed by the consultant to provide insight into the evaluation questions and criteria on a representative basis. The consultant may propose to select from a variety of probability based sampling methods, such as multi-stage sampling, probability proportional to size (PPS) or other approaches depending on their assessment of the SOW. However, Concern expects that the eventual sampling specification will provide for a 95% confidence level and 5% Margin of Error, and must align with BHA guidance, such as is contained the BHA Emergency M&E Guidance^[2]. For any qualitative data collection, including through Focus Group Discussions, Key Informant Interviews or other qualitative methods, Concern expects the consultant to propose a non-probabilistic, purposive approach, whereby the consultant will identify groups or stakeholders with a diverse range of perspectives, knowledge and experiences related to the programme and propose a purposive sampling approach towards reaching saturation against the evaluation questions. 

The selected consultants or firms will be responsible for developing a detailed methodology during the inception phase, which will include standardized data collection instruments such as surveys and interview guides. This detailed methodology, including the sampling approach and data collection tools, will be shared and approved by Concern prior to data collection as part of the evaluation inception report submission and review. The evaluation process will consist of three main phases: an inception phase involving meetings with the RAPID III team and submission of an inception report for both objectives, a data gathering phase that entails field visits, interviews, and surveys, and a reporting phase where the evaluation team will submit the final report summarizing the findings and recommendations.

As part of the evaluation, the selected consultants or firm will conduct a comprehensive analysis of a sample of sub-grants to gain insights into the complete project cycle of RAPID III and its impact. This analysis will include tracking the progress of the project from initiation to completion stage, assessing the extent to which targets have been met in terms of reaching the intended beneficiaries and the effectiveness of the response. Field visits will be conducted, incorporating various methods such as focus group discussions with affected individuals and key informant interviews, etc. The sample of sub-grants will consist of different responses, including drought, COVID-19, and flood, as well as different sectors (WASH, S&S, MPCA, etc.), diverse groups of affected people (such as displaced or non-displaced), modalities (such as cash or in-kind assistance), and types of implementing partners (international, national, and local NGOs.

The report will highlight key strengths, areas for improvement, and actionable recommendations to enhance the overall effectiveness of RAPID III in addressing the relief and early recovery needs of disaster-affected populations while building the capacities of individuals and institutions for effective humanitarian response.

By following this rigorous methodology, the evaluation will produce a high-level report that is evidence-based, impartial, and of the utmost quality, providing valuable insights to inform strategic decision-making and enhance the overall effectiveness of the RAPID III program.

The selected firm/consultants will propose an evaluation team composed of a team leader, subject matter specialists, survey specialists, and a survey team, etc.

Gender Considerations:

The evaluator(s) must give special attention to gender considerations during the evaluation process of RAPID III by thoroughly analyzing the gender dimensions to understand how the fund has addressed the specific needs and vulnerabilities of different genders during its interventions.

The evaluator should assess the extent to which gender equality and women's empowerment have been integrated into the fund's projects and activities. This involves examining whether gender-sensitive policies and strategies were in place and whether women and marginalized groups were adequately represented in the decision-making processes during humanitarian response.

Moreover, the evaluation should carefully examine the impact of the RAPID Fund's initiatives on gender roles and dynamics within the communities it serves. Particular attention should be given to understanding how these interventions have affected the lives of women, men, boys, and girls differently, both in terms of challenges and opportunities.

The evaluator should also investigate whether the capacity-building efforts by the RAPID Fund included specific provisions for gender mainstreaming among the local organizations. This entails assessing whether these organizations were equipped to address gender disparities and ensure inclusivity in their response strategies.

Deliverables

Inception Report:

The Evaluation Team will produce an inception report not to exceed 10,000 words (excluding the executive summary and annexes), setting out:

• The team’s understanding of the functioning of the RAPID III mechanism, the contexts in which the RAPID III operates and USAID/BHA’s and Concern’s mandate in managing RAPID III
• Any suggested deviations from the Terms of Reference, including any additional issues raised during the initial consultations
• The evaluation framework, second-level questions and identification of key areas of inquiry
• An evaluation matrix showing, for each question, the indicators proposed and sources of information
• A methodology, including any changes to the proposed methodology, how the comparative analysis of the funds will be conducted, details of the triangulation strategy
• The evaluation criteria to be used, including the rationale for using each evaluation criterion and, if needed, for rejecting any of the criteria proposed in the Terms of Reference
• How gender, age and other cross-cutting issues will be analyzed during the evaluation
• Data collection tools (survey, interview questions, document with the preparation of field visits and schedule of interviews, etc.)
• Description of any limitations of the chosen methods of data collection and analysis
• Other methodological limitations and evaluability issues and how they will be addressed
• Stakeholder analysis and a plan for their involvement in the evaluation process
• Data collection plan
• Draft dissemination strategy of the evaluation findings and recommendations
• Draft outline for the final evaluation report

Evaluation Report (1st Draft & Final Report)

The Evaluation Team will prepare a comprehensive evaluation report that presents the findings in a clear and accessible manner, ensuring that readers can easily comprehend the evaluation results and their interconnectedness. Considering the two main objectives of RAPID III, the report will be structured in two parts: Part I will analyze the humanitarian response aspect of RAPID III, while Part II will focus on the capacity-building component. It is important to note that the report should not merely summarize the findings of each component but provide a thorough analysis. The report should be concise, not exceeding 15,000 words or 25 pages (excluding the executive summary and annexes). It should include the following elements:

• Table of contents
• Executive summary of no more than 2,000 words
• Summary table linking findings, conclusions and recommendations, including where and with whom responsibility for follow up should lie
• Analysis of context in which RAPID III was implemented
• A desk review of documents, previous evaluations and studies relating to similar funding mechanism, and a summary explanation of how this evaluation is positioned among them
• Overview of how the RAPID III is being used in Pakistan: objectives, amounts to various response and types of activity, etc.
• Methodology summary – a brief chapter, with a more detailed description provided in an annex
• Main body of the report, including findings in response to the evaluation questions, conclusions and recommendations
• Annexes will include: (1) Terms of Reference, (2) detailed methodology, (3) analysis of RAPID III funding flows, (4) list of persons met, (5) details of all surveys undertaken, (6) details of any quantitative analysis undertaken, (7) team itinerary, (8) all evaluation tools employed, (9) bibliography of documents (including web pages, etc.) relevant to the evaluation and (10) description of selection of case studies including the Field Visit Region (Province/District, etc.) Selectivity Matrix, (11) list of acronyms

For accuracy and credibility, recommendations should be the logical implications of the findings and conclusions. Recommendations should follow logically the evaluation findings and conclusions and be:

• Categorized as a) Critical, b) Important, or c) Opportunity for learning
• Relevant, realistic and useful, and reflect the reality of the context within which RAPID III operates
• Clearly stated and not broad or vague
• Realistic and reflect an understanding of USAID/BHA and Concern and the humanitarian system and potential constraints to follow up
• Suggest where responsibility for follow-up should lie and include a timeframe for follow-up
• Small in number

Evaluation Briefs

The Evaluation Team will produce evaluation briefs of no more than 2/3 pages for the final report summarizing both objectives of RAPID III.

Dissemination and Follow-up

1. The Evaluation Team will conduct the following presentations:
2. For all deliverables, draft versions will be submitted for comments, which should be considered for the final version. Several rounds of comments may be necessary. For each round, the Evaluation Team will prepare a comments matrix, listing all comments received and explaining how they have been addressed or why not.
3. All deliverables must be written according to the USAID Style Guide^[3]. The final versions must be proofread and undergo professional graphic design. All deliverables should include relevant graphs, charts and maps to present findings and trends visually.

• Upon completion of the draft evaluation report, a validation workshop will be held in Islamabad with a presentation of the main findings, conclusions and recommendations. The presentation will be shared with the RAPID III designated person.
• Once the evaluation is completed, presentations of the main findings and recommendations will be made to various fora as decided by Concern’s management in consultation with USAID/BHA Pakistan team. The Evaluation Team may be requested to assist with these presentations.
• The recommendations addressed at RAPID’s sub-grantees will be discussed at the lesson-learned workshop to develop action plans for their implementation.
• For recommendations relating to RAPID III, a Management Response Plan will be prepared as per Concern/USAID/BHA Evaluation Policy.

Limitations

1. Due to the complex nature of the RAPID III mechanism, USAID/BHA has waived off the baseline values for sectorial indicators. However, RAPID III has conducted final verifications/assessments of each sub-award, which will help in understanding the impact of RAPID responses to different emergencies. The consultant will have to consider the indicators agreed milestones for determining performance.
2. No Objection Certificate (NOC) for the international consultant (if selected) visiting field areas in the country may take a longer time.
3. Consultants may face attribution challenges given the multitude of actors, initiatives, and external factors that contribute to humanitarian response and capacity-building efforts against different emergencies.

Implementation & Logistic arrangement

Duration

A total of 40 days’ assignment^[4], the evaluation is expected to be started tentatively in June/July 2024.  Exact schedule/plan of the consultancy will be agreed with the consultant during the signing of the contract.

Line of Communication and Reporting:

The Program Director RAPID Fund is the overall in-charge of the consultancy. Deputy Director and Monitoring and Evaluation Specialist will supervise the review

Consultant/Team Essential and desirable Expertise/experience: 󠄯

1. Post-graduate degree in Humanitarian Studies, Disaster Management, Development Studies, and/or relevant Social Sciences discipline. 󠄯 󠄯
2. At least 10 years’ experience of conducting evaluations of emergency and humanitarian programmes. 󠄯
3. Previous experience of conducting final evaluations of Umbrella Grants/pool funds in emergency and humanitarian contexts; financial analysis;
4. Familiarity with international quality and accountability standards applied in emergencies. 󠄯 󠄯󠄯 Strong analytical and conceptual skills.  󠄯
5. Experience in the use of participatory methodologies and developing equality and gender sensitive evaluation methodologies; 󠄯 󠄯
6. Experience with USAID/BHA funded projects an advantage.
7. Excellent facilitation skills, co-ordination, negotiation skills and oral and written communication skills in English (particularly report writing).
8. Competency in Equality & Gender issues;  
9. Experience in assessing organizational capacity and gaps and ability to recommend the corrective measures. 󠄯

• Excellent written and spoken communications skills in English. Knowledge of the Pakistani regional context and languages will be an advantage; Concern Code of Conduct (CCoC) and Associated Policies

Selection Process and Criteria:

The proposals will be evaluated through a competitive selection process. Following criteria will be followed.

First Stage- Technical Proposals:

-          Consultant’s profile including profile of key individuals/CVs submitted: 40 points^[5];

-          Proposals including Methodology/tools and approach: 30 points;

-          Depending on the level of competition, three consultant will be shortlisted for submission of financial proposal.

2^nd Stage- Financial Proposal

-          First lowest: 30 Points;

-          2nd lowest: 20 Points;

-          3rd lowest: 10 Points.

The selection committee will add the points of technical proposal (first stage) with the points secured in second stage to select the consultant for the external evaluation.

Note: This consultancy is open to both National and International applicants. Individuals, group of individuals as well as consultancy firms having the requisite skills/ experience are eligible to apply. International applicants having prior work experience in Pakistan with an existing Visa/NOC to work in Pakistan are encouraged to apply however, Pakistani counterparts with ability to freely move across the country would be essential.

Please fill and submit duly filled Disclaimer form to HR Department along with technical proposal

Safeguarding at Concern: Code of Conduct and its Associated Policies

Concern has an organisational Code of Conduct (CCoC) with three Associated Policies; the Programme Participant Protection Policy (P4), the Child Safeguarding Policy and the Anti-Trafficking in Persons Policy. These have been developed to ensure the maximum protection of programme participants from exploitation, and to clarify the responsibilities of Concern staff, consultants, visitors to the programme and partner organisation, and the standards of behaviour expected of them. In this context, staff have a responsibility to the organisation to strive for, and maintain, the highest standards in the day-to-day conduct in their workplace in accordance with Concern’s core values and mission. Any candidate offered a consultancy opportunity with Concern Worldwide will be expected to sign the Concern Code of Conduct and Associated Policies as an appendix to their consultancy contract. By signing the Concern Code of Conduct, candidates acknowledge that they have understood the content of both the Concern Code of Conduct and the Associated Policies and agree to conduct themselves in accordance with the provisions of these policies. Additionally, Concern is committed to the safeguarding and protection of vulnerable adults and children in our work. We will do everything possible to ensure that only those who are suitable to do consultancy work or volunteer with vulnerable adults and children are recruited by us for such roles. Subsequently, being hired as a consultant with Concern is subject to a range of vetting checks, including criminal background checking.

Interested consultants should send the following documents in a sealed envelope, clearly marked “RAPID II External Evaluation Consultancy”  and should reach Concern Worldwide Office Plot No 144, Street No 30, Sector G-9/1, Islamabad, Pakistan on or before 15:00 Hrs April 30,  2024

1.     Cover letter detailing the consultant’s/institutions suitability for the assignment and current contact information, and submit profile of the firm if applying as a consultancy firm

2.     An evaluation proposal of no more than 5 pages long (including methodology, data collection tools, previous institutional experience, timeframe etc.) The proposal should be in MS Word  format– font size Times New Roman 11 – A4 size page- margin one inch all sides –alignment justified);

3.     CV(s) of consultant (s) who will mainly carry out the assignment – each CV should at least reflect educational qualification, previous relevant experience, current location etc. and should not exceed three page MS Word  format– font size Times New Roman 11 – A4 size page- margin one inch all sides –alignment justified;

4.     At least two relevant example of previous evaluations carried out preferably in Pakistan with international NGOs (Concern will strictly ensure the confidentiality of the evaluation reports);

The Technical Proposals along with required documents could be submitted through courier/by post or in person and should reach at the specified location, within the specified deadline and in the appropriate format. The cost proposal/budget is not required at this stage, Concern will review the technical proposal/CVs/previous experience and only shortlisted consultant will be called for financial proposal.

For any questions and queries, you can send an email to [email protected]  by 15:00 Hrs April 25, 2024