Monitoring, Evaluation and Research

The MER section is a central hub for collecting, managing and updating the data for all programme sectors and projects across NRSP’s programme districts. The analysis of this data contributes to conceptualising projects, informing project design and assessing strategies and progress. Our rigorous monitoring system provides continuous feedback to the management and to other development partners. All sectors and programmes are responsible for monitoring, although MER plays the leading role. The section also designs monitoring frameworks for large projects such as those funded by PPAF, the World Bank, the ILO and other UN Agencies.

Monitoring is done both through direct interaction with communities and the programme management and through a formal process of reporting and monitoring at different levels. In communities, MER staff attend regular Community Organisation meetings, village-level committee meetings and Activists’ meetings: this interaction allows them to gain a better understanding of many kinds of community issues. These insights can then inform programme decisions.

The MER section plays a key role in the design, planning and implementation of the mainstream programme and the Projects that NRSP implements. We assist both management and beneficiaries in establishing goals and objectives and in identifying the constraints that may arise. MER also assists the management in identifying relevant monitoring and evaluation indicators for inputs, processes and outcomes. MER professionals design Log Frames, Survey forms and other tools.

Recent examples include our work for the USAID-funded Small Grants and Ambassadors Fund project and the Medicinal Plant Value Chain Project in partnership with USAID-MEDA (District Swat in KPK). For the SGAFP, MER worked closely with USAID on developing programme strategy, information systems, the staffing structure, targets for disbursement, operations manuals, the work plan and rollout plan.

Benazir Income Support Programme: National Rollout of the Poverty Scorecard Survey funded by the Government of Pakistan

From October 2010-March 2011, the MER section managed NRSP’s role as a partner in the nationwide Benazir Income Support Programme (BISP). NRSP covered the major portion of households under this programme (56% of all surveyed households) covering over 7.6 million households in 32 Districts. NRSP completed the BISP survey in five months.

BISP Pilot 2009: NRSP was selected as a partner following a pilot phase initiated in 2009. Three organisations participated in this pilot: RSPN (in partnership with the Rural Support Programmes), the Population Census Organisation (PCO) and PPAF (through consultants). RSPN and PPAF were each assigned five Districts and PCO six. 56% of the total households to be surveyed were in NRSP Districts. NRSP and the other RSPs successfully completed the pilot survey in the record time of one month. An independent evaluation by the World Bank found the RSPs’ work to be highly satisfactory. BISP then divided the rest of the country into four clusters (Sindh, Lower Punjab, Upper Punjab/AJK and KPK-GB). The remaining districts of Balochistan were assigned to the PCO and PPAF was allocated eleven districts. AHLN (an independent consulting firm) was to survey Southern Punjab, and RSPN, in collaboration with NRSP, AKRSP, SRSO, TRDP, and SRSP was to cover the three other clusters.

Strengthening Human Resources for BISP rollout

NRSP maintains a flexible organisational structure, absorbing and restructuring according to project and programme requirements. NRSP recruited 5,685 staff members to conduct the BISP survey. These staff included 4,015 enumerators, 365 campaigners and 365 monitors. All staff were hired and trained in the record timeframe of 20 days.

In all the program activities NRSP strives for inclusion of marginalized and disenfranchised community members. These include the most vulnerable, the poorest, women headed households, the elderly and children. For inclusion of the poor NRSP is using the Pakistan Poverty Scorecard (PSC) as a tool. During the Situation Analysis at a Union council level, a census of the UC is conducted using the PSC.

These staff members were then posted to Lahore, Rawalpindi, AJK, Mardan and Sindh. All were closely monitored during the first month with on the job training also provided. This strategy meant that daily progress was deliberately kept slow for the first month to streamline all systems. Each team’s progress was monitored daily by the BISP District and Divisional level offices with information from the field feeding back to the NRSP Head Office. The BISP was completed in all Districts within the agreed five months in spite of the fact that the KPK survey teams worked in the post-flood disaster hit areas. The disaster affected communications and mobility and led to the displacement of thousands of families. After completion of the data entry, BISP and NADRA announced that the survey data was 97% accurate.

REPORTING

NRSP’s MER section produces a Programme Update which contains overviews, progress and current data. Monthly Progress Reports originate at the Field Units and provide the basis for the Regional and Programme MPRs.

Our Annual Progress Reports provide cumulative data on all activities undertaken during the year. The report includes a Statistical Abstract with cumulative and annual data and the Audited Statement of Accounts. The narrative focuses on a different theme each year. Recent examples are: ‘Disaster Mitigation’ and ‘Including the Vulnerable’.

Project Completion Reports are summary documents which encapsulate the objectives and achievements of a project. They also contain accounts by community members of their experiences with the project as well as recommendations for future projects.

Case studies: When a situation warrants it MER conducts an analytical case study of a subject that requires attention. These studies may contribute to resolving issue across the programme area.

Evaluations

MER strives to support the programme by analyzing the processes, products and projects of NRSP in terms of their efficiency, effectiveness, relevance, impact and sustainability. It uses quantitative and qualitative methods of social research to answer these and many more questions. MER also conducts comparative analyses of interventions and products from time to time to identify comparative advantages. Since NRSP has a dynamic programme, MER ensures that required adjustments are made in the light of experiential learning. The evaluations are conducted either through in-house resources or by hiring third party consultants. In addition, donors also hire independent consultants to evaluate their projects.

Internal Impact Assessment/Evaluation Studies are conducted periodically by MER professionals.

Action Research Studies: MER has conducted numerous research studies since its inception. These may focus on issues such as: identifying specific aspects of poverty that NRSP can address; gender-specific issues relating to programme implementation; aspects of social mobilisation, including those relating to new kinds of interventions and new models of management to improve service delivery in under-served remote areas. Depending on the findings, the outcomes of the Action Research may include strategies for scaling up and ways and means of ensuring sustainability. Some examples are:

Social Mobilization for Empowerment (the MORE project) funded by the World Bank / PPAF Project manual (Urdu)
Microfinance for Decent Work (2009-2011) funded by the ILO
Union Council Based Poverty Reduction Programme, Mianwali. UCBPRP Evaluation
Use of the Poverty Scorecard: Implementation and Analysis Experience of the National Rural Support Programme March 2008
Community Schools Project (Programme wide)
Union Council Commercialization of Wind Power Potential in Pakistan (Pasni)
The Role of Women in the Irrigated Agriculture of Southern Punjab, Pakistan (2006-07). International Master of Science in Rural Development, University of Ghent, Erasmus Mundus Programme. Ms Gwen Willeghms.

The MER section also documents the best practices or the most innovative or unique projects implemented by NRSP including various programme level interventions designed and implemented through NRSP’s own resources. These studies include the following:

NRSP’s Owner Driven Low Cost Housing August 2011
NRSP’s Response to the Earthquake 2005
Pro-poor Water Harvesting Systems in Drought Prone Areas: A Case Study of the Karez in Balochistan, Pakistan (2003). This study was presented at the 3rd World Water Forum in Kyoto, Japan.

Situation Analysis and Baseline surveys

These studies, conducted before NRSP enters a new Union Council, provide MER with the opportunity to interact with people in new areas, as we gather information on an area’s natural, social and economic characteristics. The process makes effective planning possible by providing an opportunity to understand community dynamics and enabling local participation.

Targeting: In all the program activities NRSP strives for the inclusion of marginalized and disenfranchised community members. These include the most vulnerable, the poorest, women headed households, the elderly and children. NRSP uses the Pakistan Poverty Scorecard (PSC) as a tool to identify these community members: during the Situation Analysis a census of the UC is conducted using the PSC.

Using secondary sources, PRA methods and quantitative surveys we learn about the livelihood opportunities available to local men and women, where the ‘sites’ of power and decision making reside and who the poorest and most vulnerable people are. The Situation Analyses are also used to collect quantitative data on population characteristics, poverty levels, school enrollment, sanitation practices and on-going projects by the various levels of Government, NGOs and CBOs. It also assesses the state of education and health facilities. Examples include: