Terms of Reference (ToR) for Final Evaluation
1. General Information
Final Evaluation of the Education Sector Program Implementation Grant (ESPIG)
GPE Funded ESPIG
Program Manager- Education Sector Program Improvement Grant
40 working days
26nd Jun, 2022
2. Background Information
2.1. About Education Sector Program Implementation Grant (ESPIG)
The Global Partnership for Education (GPE)-funded Education Sector Program Implementation Grant (ESPIG) is aligned with the priorities of the Federal Government of Somalia’s Education Sector Strategic Plan (ESSP 2018-2020). The overall objective is to increase access to quality education for more than 50,000 out-of-school children; enhance the quality of primary education; and improve the capacity of the Ministry of Education, Culture and Higher Education (MOECHE) at Federal/Member States to regulate and manage the education sector. The Federal Government of Somalia Ministry of Education, Culture and Higher Education (MoECHE) is implementing the grant in collaboration with the Federal Member States Ministries of Education in Jubaland State, South West State, Galmudug State, Hirshabelle State and Benadir regional administration with CARE as the grant agent and Concern Worldwide as sub-contractor.
The main objective of the ESPIG is to improve equitable access to and quality of education outcomes for all Somali primary school children through strengthened system capacity to design evidence-driven Education Sector Strategic Plan (ESSP) reforms, and collaborate effectively with partners in their implementation. ESPIG was envisioned to maximize the potential of development investments in education in Somalia through the combination of four key principles:
- Use of evidence-based approaches to address key gaps in access, learning and retention, and inform tailored efforts to reach marginalized groups;
- Creation of conditions to implement solutions that address the intersection of multiple barriers to access, retention and learning;
- Generation of synergies with privately managed schools/networks, development actors and the private sector to ensure complementarity of efforts and leverage existing capacity/ investment;
- Capacity building of federal, state, regional and district-level actors for a cohesive, efficient and dynamic approach to the design and implementation of solutions.
The program’s five components are described below:
- Equitable access to quality education will increase through changes in three intermediate outcomes: Increased access to education for out of school children; strengthened capacity of community education committees; and community actions towards improved school safety.
- Enhanced quality of education, leading to grade-appropriate improved learning outcomes, will be achieved through the following intermediate outcomes: effective monitoring of teacher education and management policy, strengthened and harmonized efforts in teacher pre-service training; improved access to teaching and learning materials; and a strengthened assessment framework.
- Enhanced system capacity to regulate and manage the education sector will emerge from the combination of the following intermediate outcomes: (i) strengthening regulatory and monitoring system for private, community and government schools and (ii) improved capacity of education officers for planning, budgeting, policy implementation, coordination and progress tracking.
- Build a strong framework for program monitoring, accountability and communication, generating evidence to track progress, inform adaptive management and support planning and management processes.
- Effective and efficient program management, embedded within government systems at local level and leveraging the grant agent’s capacity and experience in country and globally
2.2. Program beneficiaries
Direct program beneficiaries includes: (i) about 50,000 out-of-school poor or marginalized primary children who will benefit from school inclusion grants; (ii) about 600 Community Education Committees who will benefit from training to improve inclusion, security and learning at their schools; (iii) 6,570 teachers who will benefit from teaching guides associated to the new curriculum; (iv) 240 primary pre-service student teachers who will benefit from new teacher training institutes; (v) 2,000 teachers benefitting from tailored in-service teacher training; (vi) 297,168 grade 1-8 pupils who will receive textbooks linked to the new curriculum; (vii) 150 education officers will benefit from professional development to improve skills in planning, coordination and monitoring.
3. Objective of the Final Evaluation
The Education Sector Program Implementation Grant (ESPIG) is seeking to procure the services of a consultancy company to conduct a final evaluation to assess the extent to which the stated objectives and ESPIG components were achieved (or not) in the project i.e. the extent to which the expected results of the ESPIG intervention have been achieved. The evaluation should generate relevant findings, lessons learned and recommendations to guide and inform on the best practices and strategies/approaches for improving other future phases and education programs. The final evaluation will have full participation of the project staff and key stakeholders.
3.1 Specific Objectives of the Final Evaluation
The consultant will critically evaluate the outputs and outcomes achieved under each component and how those have influenced the situation of target groups and beneficiaries. This will require critical examination of the individual activities to understand their contribution towards achievement of the results. The consultant is expected to reference changes in target groups, final beneficiaries and school conditions at the end of the project vis a vis the endline and baseline. The specific objective of the evaluation is to evaluate the outputs and outcomes achieved under each component and the project’s overall relevance, effectiveness, efficiency, validity of design, sustainability, factors affecting performance, alternative strategies and its strengths and weaknesses. In addition, the final evaluation also intends to specifically assess the uses and implementation of the second tranche of capitation grants.
Overarching evaluation questions: The final evaluation will be guided by the overarching questions below;
Table 3 : Overarching questions
- To what extent were the overall objectives and specific objectives achieved?
- What were the major factors influencing the achievement or non-achievement of the objectives?
- Were the assumptions underlying the overall program ToC valid?
- Which interventions were most effective in contributing to change relative to the personnel and associated costs?
- What have been the most and least efficient interventions?
- To what extent are the objectives of the program ToCs still relevant for the targeted groups and beneficiaries?
- Which changes in the external context or lessons in its implementation, have influenced the relevance of the ToC and how?
- How sustainable is the ESPIG as a program and what results have been achieved? And how sustainable are these results?
- What is being done to ensure the sustainability of program results and how successful are those efforts?
- What has happened as a result of the programme?
- What the major positive changes are as identified in response to the effectiveness of the interventions?
- Did the program have any unintended negative outcomes?
- Did the program intended outcomes lead to unintended positive or negative changes?
Lessons and recommendations
- What are the main lessons learnt during the program design and implementation?
- Which adjustments to the ToC should be made in the next phase of the program?
- What are the main recommendations of this component and the program in general?
Specific evaluation questions: The assessment of the program components will be based on specific evaluation questions guided by the overarching questions above. The specific evaluation questions under each component are listed below:
Component 1: Equitable access to quality education
- Provide capitation grants to target schools to increase access to education for out of school children
- Strengthen capacity of community education committees
- Community actions towards improved school safety
- What was the impact of the intervention on children’s access to school? How did the impact vary per location and gender? To what extent did the intervention result in the enrolment of children from marginalized groups?
- To what extent did the provision of capitation grants mitigate the impact of COVID-19 crisis and prolonged drought on the most vulnerable children’s education?
- To what extent did the COVID-19 crisis and drought affect the ability of schools to ensure the attendance of students benefitting from capitation grants?
- What was the impact of the school grants component on school infrastructure?
- Are community education committees better prepared to manage schools? Have they developed skills and capacities to deliver needed services?
- What was the impact of the intervention on education outcomes in targeted schools, particularly in terms of enrolment, attendance and retention?
- To what extent did the federal member states and communities in underserved areas benefit from the intervention?
- Were capitation grants implemented in an efficient and transparent manner? To what extent are community education committees and local education officials involved in their oversight? What were the main uses of capitation grants? To what extent did capitation grants increase enrolment in targeted schools? To what extent did capitation grants support increased enrolment for marginalized children and girls?
- What activities are being implemented by community education committees? Has the intervention contributed to changes in CEC activities (types of activities, support to subgroups of students and/or how those activities are being implemented)?
Given the large scale implementation of capitation grants, the assessment of this subcomponent will include an extensive review of expenditure at school level and enrolment and attendance records. This aspect of the assignment will include both an evaluation of the impact of the grants as well as an independent verification of their implementation and use.
Component 2: Enhanced quality of education, leading to grade-appropriate improved learning outcomes
- Monitor teacher education and management policy
- Construct and equip one teacher training institute
- Distribution of primary school textbooks
- Early grade assessments
- Tailored in-service teacher training
- What was the impact of the provision of primary school textbooks on learning environments?
- Are learners benefiting from increased access to teaching/learning resources? How?
- Are teachers benefitting from increased access to teaching/ learning resources? How?
- To what extent was the implementation of this component affected by the combined negative impact of the COVID-19 crisis, prolonged drought and security issues in 2020-22?
Component 3: Enhanced system capacity to regulate and manage the education sector
- Strengthen regulatory and monitoring system for private, community and government schools
- Strengthen capacities of education officers for planning, budgeting, policy implementation, coordination and progress tracking.
- EMIS system development and support
- Conduct joint review of education sector
- Support ESA/ESSP processes
- In what ways has the program enhanced fiduciary management and utilization of resources at the MOECHE and FMS MOEs?
- To what extent are the policies developed under the ESPIG, such as the private school policy and CEC training harmonization, being implemented?
- How have the various trainings, infrastructural support enabled MOECHE and respective state authorities to handle activities within their mandate effectively?
- To what extent is the new EMIS operational, compared to the previous version?
- How is the newly developed EMIS different from the previous versions? Does the system enable the MOECHE to better assess the status of the education system?
- To what extent have planning processes, such as the JRES, been supported by the ESPIG?
Component 4: Build a strong framework for program monitoring, accountability and communication
- Conduct evaluation studies – baseline, midterm study and final evaluation
- Support the MoECHE and Federal Member States (FMS) to carry out monitoring activities to verify progress against ESPIG / ESSP indicators
- Learning and communication
- Feedback and Complaints Response Mechanism (FCRM)
- How have the recommendations of the baseline/midterm survey informed adaptation and reprogramming efforts?
- Are the MoECHE and FMS Ministry of Education undertaking monitoring activities? Has support to monitoring activities enabled accountability and adaptive management?
- Are there platforms to share learning and/or communication? How is this working?
- How is the Feedback and Complaints Response Mechanism working? How can it be improved?
Cross-cutting Issues: Particular attention will be given to cross cutting issues such as special needs and inclusion, gender, vulnerable and marginalized groups, human rights, child protection and safeguarding, and health and hygiene promotion in the project. A conflict-aware, do-no-harm approach has been used to implement activities.
4. Technical Approach and Methodology
The consultancy company is expected to propose a relevant methodology to assess the intervention’s contribution to the Somali education system, responding to the evaluation questions above. In addition, the evaluation will assess the program’s relevance, efficiency, sustainability, coordination, and lessons learned**. The selected methodology should be able to show the extent to which the program achieved its intended results thus far and reached the intended target populations. The proposed methodology should also allow the program to identify and quantify the impact of contextual factors on the expected results.
4.1 Document Review
The consultant will be expected to review all relevant documentation in order to understand the assignment and the context beforehand. This will enable the consultant to gather secondary data that will complement the primary data that will be collected. CARE will provide other relevant documentation on implementation once services are contracted.
4.2. Primary Data Collection
The primary data will be collected from various qualitative and quantitative data collection methods namely:
- School survey (longitudinally tracking the schools sampled at the ESPIG baseline and midline studies);
- Analysis of school enrolment records, retention and transition records, test scores and attendance records, and observation of students’ attendance through headcounts, where possible.
- Review of school-level documentation on expenses associated to capitation grants.
- Interviews with CECs, teachers and head teachers;
- Key informant interviews with the regional and district officials, other MOE and MOECHE staff, and partner agencies;
- Case studies with beneficiaries, including but not limited to students (particularly girls) who have enrolled/ re-enrolled through capitation grants.
Other methods may be added as appropriate.
The proposed methodology should mirror the baseline and midline studies. It is expected, however, that the use of capitation grants and new enrolment associated to them will need to be verified in a larger sample of schools (minimum 210), which will overlap partially with the longitudinally tracked baseline and midline sample of 171 schools.
4.3. Data Collection, Processing and Analysis
The consultant is expected to train enumerators on data collection methodologies and tools; quality assurance’; confidentiality; ethics and informed consent. CARE Somalia will provide training on child protection and prevention of sexual harassment, exploitation and abuse (PSHEA) and require all enumerators to comply with its PSHEA and child protection policy. In addition, the consultant will be expected to supervise data collection in the field, translation, and transcription of qualitative data, ensuring quality is maintained throughout the entire process. The data collection tools will be prepared and shared with CARE before the training and thereafter piloted and revised accordingly after the pre-test exercise to ensure they are acceptable and match the program’s needs. The Consultant will then analyze the data collected, prepare the final evaluation report and present the findings, conclusions and recommendations to CARE who will then share it with consortium partners, the donor and other key stakeholders.
Deliverables under this consultancy will include the following:
- Inception report: Including the proposed methodology, data collection tools, analysis framework and a detailed work plan.
- Data collection report
- Complete datasets and syntax files used for analysis.
- Complete transcriptions of qualitative data and respective audio files.
- Draft report: Will be presented to CARE for input and feedback. CARE will give feedback within seven days of receipt of the draft report.
- Final report inclusive of CARE’s feedback.
- Summary of findings and presentation.
6. Duration and Time
The exercise is expected to commence immediately the contract is signed by all parties. The estimated deadline for completion is July 30, 2022.
Duration (Man days)
Literature review/Desk Review
Preparation of inception report, data collection tools, translation into Somali and testing/revision of tools
Travel to and from target locations
Field data and information collection
Data and information analysis
Report writing and presentations
7. Desirable Qualifications:
This assignment requires previous experience in evaluating large-scale system development programs and in particular, large-scale education programs, preferably in the region and/or in Somalia. In addition, the consultancy company should have experience in working in fragile, conflict-affected contexts. The consultant will be required to devise appropriate strategies to generate as much information as possible within the allocated period, considering the ongoing crisis and potential challenges to reach some locations.
Education and background required:
- Extensive experience in assessing large scale education programs, particularly system development/ strengthening initiatives;
- Demonstrated expertise in quantitative analysis and use of mixed methods;
- Excellent report writing skills in English;
- Adequate knowledge of the context, including the Somalia education system, policies and guidelines.
- Previous experience of conducting community-based and school surveys in Somalia.
- Ability to roll-out large-scale data collection and ensure data quality in the South Somalia context.
8. Core Values & Critical Key Competencies
- Strong analytical skills with strong ability to do editing and proof reading
- Ability to think critically and strategically in difficult conditions
- Excellent interpersonal communication, relationship building and networking skills
- Ability to work under pressure and meet tight deadlines in stressful conditions
9. Administrative/ Logistical Support
The consultancy company will submit to CARE a financial proposal covering consultancy fees, operational costs and reimbursables, in line with the methodology described in the technical proposal.
9.2 Schedule of payment
The following payments will be paid to the consultant based on an agreed upon mode of payment.
- After inception report: 30%
- After data collection is completed and datasets are received: 40% (totalling 70% of the payment)
- After the final report: 30% (final payment)
The payments will only be made when the deliverables have been assessed and approved by CARE-SOM against the set quality standards.
CARE will provide the following support towards the successful execution of this consultancy:
- Provide all relevant internal documents needed during the desk review.
- Pay for all flight costs.
- Provide transport and accommodation in the field during the activity
- Provide stationeries and supplies required during the activity
- Provide linkages with key stakeholders and actors at all levels to facilitate key informant interviews.
- Pay consultant’s fees upon satisfactory completion of the assignment.
10. Conditions of Work
The ESPIG MEAL Lead will coordinate activities related to this consultancy in Somalia. CARE USA’s Senior Research and Learning Advisor will provide technical assistance to the evaluation process. The MEAL Lead and the Senior Research and Learning Advisor will review and approve the deliverables. The consultant will be required to abide by the organization’s applicable rules and regulations, code of conduct and child safeguarding policy. CARE upholds the principle of working independently of political, commercial, military, or religious objectives. CARE has a zero-tolerance approach toward sexual harassment, exploitation and abuse, and child abuse.
11. How to apply
All interested consultants/firms are requested to write an expression of interest following the attached EOI format ONLY by email to: SOM.Consultant@care.org. Please indicate ‘Final Evaluation – Education Sector Program Implementation Grant’’ as the subject heading. Application deadline 27 JUN, 2022. Any canvassing will lead to automatic disqualification.
How to apply
All interested consultants/firms are requested to write an expression of interest following the attached EOI format ONLY by email to: SOM.Consultant@care.org. Please indicate ‘Final Evaluation – Education Sector Program Implementation Grant’’ as the subject heading. Application deadline 30 JUN, 2022. Any canvassing will lead to automatic disqualification.