Monitoring and Evaluation system - Roma Education Fund

A GOOD START – Monitoring and Evaluation

In order to design, plan and then measure success across a project it is essential to collect and use statistical information. In the case of the Roma integration projects there is very limited data for the region and where it exists it is mostly fragmented and not comparative. For ‘A Good Start’ a comprehensive monitoring and evaluation system was designed in order to capture the initial situation, and to determine the effect of the project on its beneficiaries. A monitoring team made up of Roma Education Fund (REF), Slovakian Governance Institute (SGI), UNDP and the World Bank developed the framework and trained the partner organizations to use the different tools. As the project covers 16 locations across 4 countries, the design of the monitoring and evaluation system had to be quite innovative to allow for comparison and lessons learned across the different places and situations.

The main tools include: a Community Assessment establishing a baseline of information for each locality on the families and the early childhood education and care services available; a Household Survey focusing on parents’ attitudes and behaviours on education and care with basic socioeconomic data; and a Database which incorporates the household survey, information on families’ participation in project activities, children’s attendance at pre-school amongst others. There is also the Continuum for Assessing Caregivers (a classroom observation tool) to determine the standard of teaching in the pre-schools, including attention to diversity issues. At the end of the project groups of stakeholders including municipalities, teachers and parents, will be broken into Focus Groups to provide their feedback on the process and the impact of the project on the Roma parents and children. As extensive quantitative and qualitative information is being collected within ‘A Good Start’ the analysis should prove very useful for increasing understanding on Roma early childhood education and care issues. A guide will be produced to share information on the data collection processes.

In addition monitoring occurs throughout the project with many monitoring visits undertaken. Roma Education Fund officers visit the project implementing NGOs and also the project locations regularly. Three times during the implementation a team from REF with some representatives of SGI and UNDP visits the local NGOs, project locations, and other important key stakeholders to check that everything is on track towards achieving objectives. It offered a good opportunity to acknowledge successes to date and work with partners to improve communication and overcome challenges in implementation.

The Community Assessment is a data-table which established a baseline of information on early childhood education and care in each project locality. The majority of the information is quantitative. Its purpose is to help determine gaps in early childhood education and care provision, and thus what is needed to improve the situation, and it will also help measure the results of the project. Partner organisations collected data prior to the beginning of the project as part of the needs assessment, and again at the start of the project (Summer 2010). They will complete it one last time at the end of the project (by May 2012). The sources of information for the Community Assessment include local authorities, the school inspectorates, the school directors and staff, kindergarten and crèche directors and staff, and local Roma leaders and/or leaders of local Roma NGOs, and Roma parents.

The Community Assessment includes information on the numbers of Roma and majority population in each locality, the numbers of Roma children of eligible age to attend pre-school and primary school and those that are attending in practice. Information is provided across each age group and grade. It gives information about the number and types of education facilities in an area, who maintains them (e.g. state/ religious/ private) and what capacity they have. Information is also included on the numbers of segregated classes and schools, and the number of Roma children attending these (compared to official figures for children who have been assessed as having special needs). The Community Assessment includes information on the numbers of children commuting to school or to access care services. It also captures data on the availability of health care in the community – such as pre-natal, anti-natal, immunizations and logopaedic services. It provides numbers of parents involved in school parents’ associations and clear information on the numbers of children targeted through ‘A Good Start’.

The Household survey is a questionnaire that captures basic household information, and provides insight into the education related expectations and attitudes of the Roma parents. It helped establish the situation early in the project (November, December 2010) and will be applied again at the end of the project (May 2012). The first application of the survey was intended to help inform the design and delivery of the activities in the project that are directed towards the parents. The second application will help determine the impact of those activities on the parent’s attitudes and behaviors. The Household Survey is applied to the parents involved in the project (where they or their children are recurrent beneficiaries participating in a number of activities or activities that are ongoing, such as parenting education). The 30 – 40 minute survey was tested in October to ensure that it captured the information as intended, in a sensitive manner. There are yes/ no answers, multiple choice and open-ended questions. It was voluntary, and the Roma parents could choose not to respond to any questions as they preferred. Below is a table illustrating the number of families that are expected will be recurrent beneficiaries.

Table of number of AGS recurring beneficiaries

The mother or the main primary caregiver (when it is not the mother) is interviewed by the partner organisation’s community mediator. She/ he is asked how many children they have, their ages and gender, and what education they and their children have. She/ he is asked about the language used in the household and by their children. Socioeconomic questions are asked about the facilities in the home and employment, and there are questions on identity document and the extent the community lives in an integrated or segregated community.

There are a number of questions in the Household Survey aimed to assess the parent’s views about pre-school/ kindergarten and education. These include some questions specifically on her/ his views about girl-child and boy-child education. There is a section which includes questions on health care, both on attitudes and practices such as immunization and breast-feeding. The mothers or primary caregivers are asked also about their views and practices regarding learning in the home, such as playing and drawing, and teaching letters or numbers. There are a number questions regarding the children’s attendance at creche, pre-school or primary school, as applicable to their age-group. These questions are designed to provide some information on barriers to access and integration but also positive experiences. The questionnaire also captures information on the teachers’ communication to Roma parents about their child or children’s progress. There is a short child assessment section for children aged 3-7 years, aimed at establishing how well they are doing in terms of their education, emotional and physical development. There are questions on reading, writing, identifying letters, climbing the stairs amongst others. As well as capturing information, questions like these should have the added benefit of getting the parents to think about these issues and their importance.

 With Slovakian Governance Institute taking the lead a children’s database was designed and developed especially for ‘A Good Start’ project. The database captures all the information from the household survey question by question in a very straightforward format. It also provides information on participation of families in activities and on attendance and children’s scores on relevant tests, such as those on school readiness. All findings are illustrated by graphs and pie charts, to give an overview of all the beneficiaries in the project, and location by location. Partner NGOs were given extensive training led by SGI and UNDP on undertaking the survey and also on inputting the data. It is hoped that the partners will also be able to use the information to help with the design of other projects. As new families come into the project the household survey will be applied to them and the results will be entered in the database. Analysis of the information is underway at the moment so results should be made available shortly.

See attached REF’s privacy policy on the usage of personal data in the ‘A Good Start’ project.

 Useful links/ publications
UNDP Vulnerability profiles of Roma population in Roma decade countries http://vulnerability.undp.sk/
The Importance of Data for promoting the Right to Education, State of the World’s Minorities 2009, Minority Rights Group International http://www.minorityrights.org/7948/state-of-the-worlds-minorities/state-of-the-worlds-minorities-and-indigenous-peoples-2009.html
OSI International Comparative Dataset on Roma Education 2008 http://www.romadecade.org/5050