Terms of Reference (TOR)

Project End Evaluation

 

Project:	Strengthening Resilience through Disaster Risk Reduction, Water Management, and Sanitation in Rural Peripheral Areas in Rajanpur/Punjab. (Rajanpur Integrated DRR, WASH and Livelihood Project)
Donor	BMZ
Project Start Date:	01-May-2021
Project End Date:	31-October-2024

 

1. 1.     Introduction/Background.

 

The Integrated Disaster Risk Reduction (DRR), Water, Sanitation, and Hygiene (WASH), and Livelihood project in Rajanpur was initiated as an extension of the Humedica-BMZ funded initiative titled "Strengthening Resilience through Disaster Risk Reduction, Water Management, and Sanitation" in the rural regions of Rajanpur, Punjab. This strategic project was designed to foster long-term development in the area, with support from the Federal Ministry of Economic Cooperation and Development of Germany through Humedica International in Pakistan.

The project covered 40 villages across four Union Councils in Tehsil Jampur. Launched in May 2017, it spanned a period of 42 months, concluding in December 2020. The primary focus of the project's interventions was on WASH activities, with additional efforts directed toward disaster preparedness and livelihood support.

Despite the challenges faced during the project period, including several instances of flooding that impacted the targeted areas, the interventions aimed to break the cycle of vulnerability among the population. Building on the experiences and lessons learned from the initial phase, the follow-up integrated project concentrated more explicitly on disaster management. This current phase targets 30 villages, where all aspects of the project are being implemented in full. Additionally, the 40 villages from the previous project are also addressed with specific measures for areas that had not been adequately covered or lacked intensity in the prior initiative.

Key highlights of the project include planned vocational training, the establishment of early warning systems, and the promotion of animal husbandry. The initiative also provides opportunities for residents in the predecessor project villages to deepen and consolidate their knowledge through refresher courses, counseling, and referrals on relevant topics. Given the isolation of the targeted areas,

 

 

low education levels, and socio-cultural barriers—particularly for women—this "aftercare" is essential for ongoing community support.

Support for the district's disaster risk reduction institutions, such as the District Disaster Management Office (DDMO) and Rescue 1122, has enhanced the operational capacity of these agencies, positively impacting the entire district population by promoting preparedness and risk reduction measures.

The rural communities in the western regions of Rajanpur district are significantly isolated and marginalized from essential government services, including education, healthcare, and transportation. Their geographic vulnerability places them at high risk for flash floods, leaving them largely defenseless against such disasters. These frequent flood events often escalate into catastrophic situations, severely impacting the livelihoods and lives of local residents. The absence of protective measures and inadequate self-help capacities greatly limits the resilience of these communities to disaster events, hindering overall socio-economic development.

Furthermore, these village communities face challenges such as limited access to water for personal use and agriculture, coupled with a lack of basic hygiene knowledge. Reliance on less efficient agricultural practices and inadequate access to essential government services exacerbate the poverty situation in the region.

The target population for the current project comprises residents from 70 villages within the Union Councils of Harrand, Wah Lashari, Lundy Syedan, Noor Pur Manjhowala, Tibi Lundan, and Bhakharpur in the Jampur Sub-District of Rajanpur District. The estimated population in these areas is approximately 9,800 people, calculated based on an average of 140 individuals per village (with an average of 20 households per village and 7 people per household). The community predominantly consists of smallholders, tenants, and wage laborers, whose income sources are primarily derived from fragile crop and livestock farming, largely dependent on rain-fed agriculture due to inadequate irrigation access.

 

1. 2.   Key Findings of Baseline and Recommendations:

 

Findings related to Livelihood (Kitchen Gardening, Agriculture practices, Livestock and Vocational trainings)

Overall, out of 361 respondents, 234 (65%) reported experiencing difficulties in meeting their household food needs at some point during the past year. When asked about food shortages, many respondents indicated that these challenges were most pronounced during the summer season, prior to the monsoon. Scarcity of water during this period negatively impacted the productivity and yield of crops cultivated by the targeted communities. Although the common practice is to prepare three meals a day, only 71% of individuals in the target area managed to cook two meals daily, highlighting a shortage of resources.

Farming is the primary occupation of the target population; however, 77% of respondents reported that they have never engaged in kitchen gardening at the household level.

Furthermore, all respondents stated that they have not received any training in kitchen gardening from humanitarian organizations or government agencies.

The survey findings reveal that 96% of respondents have never conducted soil testing on their cultivation land, and 2% expressed uncertainty about the practice. Additionally, 97% of respondents were unaware of the seasonal calendars provided by the agriculture extension department for sowing and harvesting crops. Notably, 100% of respondents indicated that they have never attended any training on effective agricultural practices.

The Deputy Director of the Agriculture Extension Department noted that during field visits, local communities did not fully cooperate. This contrasts sharply with the baseline survey data, where 93% of respondents reported that no representatives from the agriculture department had visited their village to offer agricultural assistance. Respondents further indicated that government officers typically visited large landholders, providing them with support while neglecting smaller or subsistence farmers.

The survey also found that 45% of respondents do not have their livestock vaccinated. There exists a significant misconception among the targeted villages, with 70% of respondents believing that vaccination is only necessary after their livestock fall ill. This lack of knowledge and awareness is further underscored by the fact that 100% of respondents reported they have never attended any livestock management training. According to a local livestock officer, most people in the targeted areas lack knowledge about seasonal diseases and their treatments, often resorting to traditional methods for animal care.

In terms of vocational training, baseline survey findings show that 52% of respondents or their family members expressed interest in acquiring new skills. However, 63% indicated that they do not allow females in their households to pursue vocational training due to various barriers, including cultural restrictions, long travel distances, prohibitions against women working, lack of permission for women to travel alone, absence of vocational training centers nearby, and time constraints due to domestic responsibilities.

Findings related to Disaster Risk Reduction (DRR)

According to the survey, 92% of respondents identified flash floods and heavy rainfall during the monsoon season as significant hazards that they frequently experience in their area. In response to prolonged periods of food shortages due to these disasters, 56% reported selling personal goods and assets, 32% stated they migrate to other regions for labor work, 11% cited visiting moneylenders, and 1% mentioned engaging in off-farm activities.

Additionally, respondents highlighted the absence of an early warning system or any mechanism to inform them about impending disasters. Findings from the baseline survey indicate that 100% of respondents have not received any skills or training related to disaster response or first aid from any organization recently. Furthermore, there are no existing committees in their villages to respond to disasters.

Focus group discussions (FGDs) with both male and female participants revealed that the vast majority of community members (98%) are unaware of any local government department involved in disaster management. When further probed, FGD participants expressed that they do not know whom to contact for emergency assistance in the event of a disaster, and none of them have had a focal person from any government department visit to share information or knowledge regarding the prevailing disasters in their area.

The findings from the FGDs clearly illustrate a significant gap in coordination between the targeted communities and local government departments in disaster preparedness and response.

Findings related to Water and Sanitation Hygiene (WASH)

The survey findings indicate that 56% of respondents rely on canal water as their primary source of drinking water, which they store in small mud ponds. Additionally, 40% of respondents use hand pump water for drinking purposes, while the remaining 4% utilize alternative sources, including tube wells and water filtration plants. For domestic use, 58% of respondents also depend on canal water stored in mud ponds (katcha ponds), and 41% use water from hand pumps located in their villages, with only 1% using tube well water for domestic purposes.

Regarding water purification, 66% of respondents do not employ any method to filter their drinking water. Among those who do filter, 17% use cloth, 11% allow dirt to settle, 3% boil the water, and 2% use alum (phitkri) for purification.

In the targeted area, the responsibility for collecting water primarily falls on women. The findings reveal that 26% of respondents report it takes them more than 30 minutes to collect water, while 14% say it takes 25-30 minutes, 11% report a collection time of 20-25 minutes, 14% say 15-20 minutes, 21% state it takes 10-15 minutes, and 14% indicate it takes 5-10 minutes per trip to collect water and return home. Of the respondents, 89% reported washing their water storage containers; among these, 80% simply rinse them with water, 5% use ash or clay, and 4% use soap.

The baseline survey findings reveal that 81% of respondents and their family members practice open defecation. Furthermore, the results show that 52% of respondents dispose of their garbage on the ground around their homes, while 15% throw their waste far from their homes. This practice poses health risks by creating a favorable environment for germs to spread via flies. Additionally, 28% of respondents burn their waste, which is also harmful to the environment.

In the two weeks preceding the survey, 52% of respondents reported that their children experienced diarrhea. On average, respondents spent Rs. 1,811 last month on treatment for water-related diseases. Survey findings further indicate that 34% of respondents only wash their hands with water, 17% use water and sand, 3% wash with water and ash, and 46% wash their hands with soap.

 

1. 3.     Project Outcome/outputs

3.1 Outcome includes:

70 village communities across 6 union councils in the Jampur subdistrict have demonstrated increased resilience to disaster events and have made significant improvements in their livelihoods.

3.1.1 Outcome indicators:

• 70% of the surveyed target population, comprising approximately 360 households in 15 villages, report feeling safer against disaster risks and can identify at least two local disaster management measures.
• Local disaster risk reduction (DRR) plans were implemented in 90% of disaster events that occurred in the 30 new target villages during the project duration.
• Early warning systems were established in 40 target villages from the previous project and were utilized in 90% of the disasters that occurred during the project period.
• Ninety percent of the target population in the 30 new target villages now have access to safe drinking water via hand pumps.
• An Operations & Maintenance Committee is functioning for all water intake locations, and 80% of users regularly pay their user fees.
• 70% of the surveyed residents (approximately 360 households in 15 villages) have reduced their expenditures on medical treatment for water-related illnesses, such as diarrhea, by at least 20% compared to the baseline data collected at the start of the project.
• 60% of surveyed participants in skills training (120 out of 200 individuals) report a 15% increase in their income as a result of the training provided by the project.
• Among the surveyed members of the target population who participated in skills training (100 out of an expected 300 participants), there is an average reported increase in earnings of at least 20% compared to baseline income levels established at the start of the project.
• 70% of the 300 surveyed participants in the project’s livestock vaccination campaign (which reached approximately 1,050 people) are able to identify where and how to access vaccinations for their livestock in the future.
• 70% of households (approximately 600) that gained access to kitchen gardens report a significant improvement in food availability. A survey of 250 participating households will be conducted.

    

 3.2 Outputs:

• 70 village communities possess the knowledge, skills, and resources necessary to effectively address disaster risks and situations.
• 30 village communities have access to high-quality drinking water (pollution-free groundwater with acceptable levels of salinity) and have adopted appropriate hygiene and sanitation practices, resulting in a notable reduction in the incidence of common diseases.
• The food security and livelihoods of 70 village communities have been significantly and permanently enhanced through the utilization of artisanal skills, as well as skills in agriculture and livestock farming.

 3.2.1: Output Indicator-1:

• A minimum of 3,800 individuals from 30 new target villages participated in training sessions focused on disaster management and the development of local disaster management plans.
• 70% of the surveyed participants demonstrated sufficient knowledge regarding the provisions of local disaster preparedness and management plans.
• Early warning systems have been established in 70 target villages.
• Responsible individuals in these 70 target villages can identify contact persons within state disaster management and are able to reach out to them in times of need.

 

3.2.2 Output Indicator-2:

• 80% of surveyed beneficiaries, representing approximately 360 households in 15 villages, require less than 30 minutes to obtain their daily supply of drinking water.
• 60% of households across all 30 target villages have constructed and are actively using latrines.
• 60% of surveyed beneficiaries, approximately 360 households in 15 villages, can correctly describe at least three out of five steps of proper hand hygiene.
• 80% of students in the beneficiary schools can demonstrate the hygiene practices they have learned.

 

3.2.3 Output Indicator-3:

• 90% of approximately 600 training participants have established and are actively maintaining kitchen gardens.
• 50% of farmers who participated in training sessions are implementing improved farming methods or cultivating new crops.
• 75% of livestock owners whose animals were vaccinated under the project report a reduction in disease incidence among their livestock.
• 75% of participants in vocational training programs have applied their knowledge by starting small businesses or engaging in wage work within their respective fields.

 

1. 1.     Final Evaluation

The project has been implemented from 1^st may 2021 to 31^st october-2024 in 2 phases. This is a mandatory final project evaluation required by the project donor Humedica International-BMZ Fund for the purposes of assessing the effectiveness, relevance, efficiency, coherence, sustainability, and impact of project initiatives. 

 

1. 2.  Scope:

 

The evaluation will cover the entire project duration, from October 1, 2024, to October 20, 2024. During this period, the evaluation will assess project outcomes at all levels, including result-based management, beneficiaries and project stakeholders, timeline, budget, inputs, and resources. The impact assessment will provide a comprehensive evaluation of the project's interventions in the targeted villages, focusing on the overall impact and relevance of the project. It will analyze the effectiveness of project management and activity implementation in terms of budget consumption and efficiency.

The evaluation will closely examine the results achieved, the coordination with local line departments, and the level of involvement of stakeholders and beneficiaries in the project interventions across 40 previous villages, as well as 30 new ones.

Additionally, the assessment will identify challenges faced and best practices utilized for future engagement. The review will include feedback from the target group or beneficiaries, representing a diverse group that includes males, females, children, vulnerable individuals, minorities, transgender individuals, and the elderly from the target areas.

Overall, the impact assessment will provide valuable insights into the strengths and weaknesses of the project, enabling informed decisions for future interventions and ensuring sustainable positive impacts on the communities in the 40 previously targeted villages, followed by the 30 new ones.

 

2.1 : Objective of the study:

 

    The objectives of the evaluation are summarized below:

 

• To evaluate the project in terms of its effectiveness, relevance, efficiency, coherence, sustainability, and impact, with a primary focus on assessing the expected results and outcomes.
• To identify key lessons learned and potential practices for future learning.
• To assess challenges encountered, highlight best practices, and document outcomes for future processes.
• To provide recommendations for similar future interventions as well as for new initiatives in the targeted areas.

 

2.2  Evaluation Questions

 

The evaluation will be guided by the following questions by keeping DAC criteria in focus.

 

Sr.#	Evaluation Criteria	Mandatory Evaluation Questions
1	Relevance	Was the project relevant to the identified needs of the target beneficiaries and the context? Were the project activity and strategy (including its assumptions and LFA) realistic, appropriate and adequate to achieve the results? Have there been changes in the planned measures? Were some measures replaced by others or did some measures have to be intensified? Has the composition or size of the target group changed? Has the project region been expanded? Reduced? Diversified? Have the changes in the planned measures had an impact on the achievement of the objectives? Has it become clear during the course of the project that objectives have to be amended because they cannot be achieved as planned in the application? Have the measures perhaps been modified based on changed objectives for 40 old and 30 new villages?
2	Effectiveness	How appropriate were the management process in supporting delivery of project expected results? Were the project activities developed as per the core humanitarian standards and resilience needs?   How effective were the strategies and tools used in the implementation of the project? To what extent were the intended project outcomes and outputs achieved and how? To what extent did the project reach the targeted beneficiaries at the project goal and outcome levels? How many beneficiaries have been reached? To what extent has this project generated positive (or negative) changes in beneficiaries of targeted villages?   These questions will apply both in 40 old and 30 new villages.
3	Efficiency	How far the results achieved did justified the cost incurred - were the resources effectively utilized? Did project activities overlap and duplicate other similar interventions (funded nationally and /or by other donors? Are there more efficient ways and means of delivering more and better results (outputs and outcomes) with the available inputs? Could a different approach have produced better results? How efficient and timely has this project been implemented and managed in accordance with the project proposal? How efficient is the record keeping, reporting and data management of the project activities in 40 old and 30 new villages. These questions will apply both in 40 old and 30 new villages.
4	Sustainability	Are there any plans and or strategies to sustain the gains after the project interventions? How effective were the exit strategies, and approaches to phase out assistance provided by the project? What are the key factors/areas that will require addition support/attention in order to improve prospects of sustainability of the project outcomes and the potential for replication of this approach? To what extent are the existence of institutions and their ability to function ensured even after the project has been completed? How will the acquired knowledge be passed on? Who will bear the follow-up costs (for example, personnel, repairs and maintenance costs) in 40 old and 30 new? These questions will apply both in 40 old and 30 new villages.
5	Impact	What are the unintended consequences as a result of the project activities? Are there any significant changes in the context as a result of the project intervention? Is the institution/facility regularly used by the target group? Are the committee structures (VDCs, O&M committees etc. are well supported and encouraged in their work by their social environment? If the project aspires to achieve outcomes at the intermediate (meso) and large scale (macro levels), has this already generated additional interest and been emulated by others? What is the reaction of local authorities and other state representatives? These questions will apply both in 40 old and 30 new villages.
6	Learning and Replicability	What are some of the key lessons learned as a result of this project that can be shared and replicated? What are the recommendations for similar future interventions? What are the recommendations for future interventions (for future project)? Which interventions were particularly popular or unpopular among the target group, and which ones encountered difficulties? Were there important/less important measures or key impacts for achieving the objectives? These questions will apply both in 40 old and 30 new villages.

 

1. 6.     Evaluation Methodology:

The proposed methodology for the endline evaluation will combine qualitative and quantitative data collection and evaluation techniques. The methodology will be developed by the consultant, as well as all relevant tools and presented in the inception report. The data collection should include the use of a number of approaches to gain a deeper understanding of the outcomes of the project, including,

• Desk-base review: The evaluation will review relevant project documents and content produced before and during project implementation including the project proposal, work plans, project progress reports, annual project reports and other documents produced by or associated with the project.
• Survey: Data will be collected using data collection tools at household level with the sample size of 360 individuals(baseline) endline will cover 70 targeted villages including 40old & 30 new.
• KII Interviews: In addition to the desk review, the evaluation will also conduct interviews of key project stakeholders using a structured methodology developed by the consultant.

 

ü  District Disaster Management Authority

ü  Soil Conservation Department

ü  Livestock Department

ü  Agriculture Extension Department

ü  Public Health Engineering Department

ü  Social Welfare Department

ü  NGO/INGO

ü  PMS Staff/HO Staff (if required)

 

• Focus Group Discussions (FGDs):

Data will also be collected using focus groups discussion with the key project stakeholder. The details of FGDs are below.

 

ü  10 FGDs will be conducted with four Village Development Committees (VDCs)

ü  6 FGDs will be conducted with two CERTs

ü  10 FGD will be conducted with the DRR Forum

ü  3 FGD with female groups

 

 

1. Evaluation Timeline: The data collection phase in the field is to be confirmed between the consultant and PMS but ideally would start in first week of October 2024 with the final draft report deadline to PMS by the last week of October 2024.

Sr.#	Deliverables	Detail	Timeline
1	Proposed work plan and tools	Proposed work plan and evaluation questionnaires to be submitted within 02 days following the official initiation of the evaluation.
2	Fieldwork	Training of enumerators, testing of tools and final validation
3	Data Collection	Briefing, deployment of enumerators, data collection and post field work debriefing.
4	Presentation of Data (excel sheets)	Lead Consultant to present analysed summary of field data before the writing the first draft report. This will allow for review, questioning and field follow up and cross validation of the data.
5	Draft of Evaluation report	The evaluator must submit draft report for review and comments by all parties involved after analysis of the field data. Both the PMS management members and other related stakeholders in the evaluation must review the draft evaluation report to ensure that the evaluation meets the required quality criteria.
6	Final Report	Report will be submitted 4-5 days after receiving comments from the PMS management members and other related stakeholders. The content and structure of the final analytical report with findings, recommendations and lessons learnt covering the scope of the evaluation must include the following: • Executive summary • Introduction • Description of the evaluation methodology • Situational analysis with regard to the outcome, outputs • Analysis of opportunities to provide guidance for future programming • Key findings, including best practices and lessons learned • Conclusions and recommendations • Appendices: Terms of reference, field visits, people interviewed, documents reviewed.

 

 

1. 8.     Management and Logistics

The Lead Consultant report to MEAL department who will not have control over the professional work of the evaluators. The Project Manager along with MEAL office will provide technical guidance (guidance to reach targeted villages, stakeholders and key informants) for the evaluation to ensure the independence of the evaluation process. Logistical arrangements will not be provided by the PMS and is included in their (consultants’) budget.

 

1. 9.     Required Qualifications and Expertise for Lead Evaluator

      The Evaluator shall have the following expertise and qualifications:

 

ü  At least Master’s degree in Public Policy, International Development, Development Economics/Planning, Economics, International Relations/ Diplomacy or any other relevant university degree.

ü  Extensive expertise, knowledge, and experience in the field of aid effectiveness and aid management process related issues.

ü  Experience of project formulation and evaluation and practical experience of end of project evaluation in South Punjab/Sindh or in similar context; At least 08 years of experience in working with international organizations, donors and evaluating projects.

ü  Past experience in conducting surveys, especially evaluations of German Donors funded projects. (Optional)

ü  Excellent written and verbal communication skills in English.

ü  Experience in Saraiki, Balochi, and Punjabi languages will be considered an advantage.

ü  Offers will be evaluated on these criteria as well as the quality of the technical offer (proposed methodology, capacity to mobilize qualified personnel in the field, etc.) and the soundness of the financial offer.

 

1. 10.  Method of Application:

Interested, qualified and experienced person, group of persons or firms must submit an offer via sealed envelope on or before 23^rd September 2024 in a sealed envelope, outlining availability in line with the approximate timeline. The cover letter should be addressed to:

 

Pak Mission Society (PMS), Operations department, House no 333, Street no 29, G-14/4 Islamabad.

    

The Offer should contain: (a) a technical offer and (b) a financial offer, comprising:

 

1. A.    Technical offer:

• Up to date CV of the lead consultant/evaluator (showing education and expertise).
• Technical proposition detailing proposed methodology and resources needed (max 3 pages).
• An example of a report from similar work which demonstrates evidence of the skills and experience required and a list of past evaluation produced by the lead consultant/evaluator.
• The Consultant/Firm must be legally registered with the Federal Board of Revenue (FBR).

1. B.    Financial offer:

• A list of all expenses expected to be incurred by the consultant including a daily rate.

• Please note: the following offers will be automatically rejected:

 

• Any offer that is not properly sealed and marked.
• Any offer submitted via email.
• Any offer received after the due date.