Manual for the evaluation of humanitarian aid

European Community Humanitarian Office, Evaluation Office 1999

(http://www.reliefweb.int/rw/lib.nsf/db900SID/LGEL-5CYGNQ?OpenDocument)

 

 

The big crises of the nineties (in Iraq, Bangladesh, Africa, the former Soviet Union, Albania, former Yugoslavia) required donors and relief organisations to put in an unprecedented and sustained effort. The European Commission played an important role in this concerted effort, and sought to learn from the experience it gained and from the problems it encountered.

Many of the problems that arose were caused by the Commissions structures and resources being ill suited to new requirements. In a bid to solve this problem, the Commission set up a dedicated department to manage all aspects of its emergency humanitarian aid from 1 March 1992.

The department was called the European Community Humanitarian Office (ECHO), and was made responsible for managing humanitarian aid for the benefit of the populations of non-Community countries affected by natural disasters or exceptional events requiring a rapid reaction and/or accelerated procedures .

The creation of ECHO was intended :

to bring under one administrative umbrella (and thus have uniform management of) the expertise needed to cope with emergencies and to put the appropriate procedures in place ;

to expand the Communitys presence in the field by building the capability to intervene at various stages: identifying needs, mobilising intervention teams and equipment, monitoring and verification and ex-post evaluation;

to improve coordination with the Member States, other donors, nongovernmental organisations (NGOs) and specialised international agencies by encouraging combined operations and the signing of framework contracts;

to achieve easier mobilisation of the resources needed in large-scale crises.

By setting up ECHO, the Commission was also seeking to increase the European Unions visibility by making the public in the EU and other parts of the world better informed about EU humanitarian aid activities.

ECHOs responsibilities cover :

emergency humanitarian aid for the inhabitants of non-Community countries;

emergency food aid for the inhabitants of non-Community countries;

mobilisation of relief and intervention teams;

disaster prevention and disaster preparedness;

coordination, information, financial matters and legal matters.

 

 

Definition, principles and objectives of evaluation

The most appropriate way of summarising ECHOs view of evaluation is to say that an evaluation is an independent and objective survey of the relevance, effectiveness, efficiency, impact and viability of a humanitarian intervention, in order to learn lessons from experience.

ECHOs main aims of evaluation are :

to improve the management of the interventions to achieve optimum use of funds and other resources ;

to learn from experience in order to improve the results of humanitarian aid interventions ;

to respond to the requirement for reporting and improve transparency.

ECHO uses the following key criteria for its evaluations :

Relevance : assessment of the objectives of an intervention, particularly regarding their justification in the light of problems and needs.

Effectiveness : the degree to which the objectives of the intervention are fulfilled.

Efficiency : a measure of how well the resources are used to produce achievements and results.

Impact : effects attributable to an intervention.

Viability : the degree to which the desired effects of an intervention last beyond its end.

Evaluation is intended to make it possible to learn from the experience of past interventions. Rather than being primarily a means of checking implementation, it should be an aid to decision making. It is an integral part of that process.

An evaluation is not :

a scientific study : scientific studies are the result of research which probes the extent of knowledge. It usually covers a very specific subject area.

an audit : audits are intended primarily to check that programme resources have been used according to the rules. Audits traditionally involve examination of accounts.

monitoring: monitoring in the field is a continuous process coinciding with the implementation of the programme and is intended to correct any deviation from the operational objectives. Monitoring will often yield information which can be used during evaluations.

 

Varieties of evaluation

ECHO focuses on the following varieties of evaluation:

- Ex-ante evaluation : an analysis of a situation which describes and quantifies problems in a particular area, sets out objectives and looks at the appropriateness of proposed measures and ways of translating them into action. This type of evaluation should propose indicators that can be used to gauge the success of an intervention.

- Intermediate evaluation : an analysis of an intervention performed while it is being implemented. It focuses on the relevance of its operational objectives relative to its overall objectives, and on matters relating to implementation and management. It describes what the intervention has achieved and what its initial effects have been, using information available. This type of evaluation is carried out by outside evaluators recruited by ECHO.

- Ex-post evaluation : an evaluation carried out after an intervention has ended. It is designed to assess the interventions results and consequences, and to see if any lessons should be learned.

The evaluator

The quality of an evaluation rides on the professional capabilities of the evaluator performing it. The choice of evaluator is a vital matter.

The following factors are considered :

operational experience of humanitarian aid;

expertise and experience in the evaluation of humanitarian aid;

independence from the parties involved ;

ability to work in high-risk areas ;

ability to work to tight deadlines.

The quality and credibility of the evaluation would be preserved by someone with these qualifications, coupled with common sense, impartiality and integrity.

ECHOs intermediate and ex-post evaluations are carried out by independent outside consultants.

 

The terms of reference (TOR)

The terms of reference set out the boundaries of the evaluators mission, the issues to be considered and the evaluation timetable. They allow those commissioning the evaluation to express their needs and the evaluator to have a clear idea of what is expected of him. Well-defined terms of reference are vital.

> They should include the following :

the grounds for the evaluation;

the future uses and users of the evaluation;

a description of the intervention to be evaluated ;

the scope of the evaluation;

the main issues to be covered by the evaluation;

the methods to be used to gather and analyse information;

the plan, structure and budget of the evaluation;

selection criteria for outside evaluators ;

the proposed structure for the final evaluation report.

 

 

ECHOs Evaluation Unit is responsible for guiding the course of the evaluation:

The Evaluation Unit draws up the terms of reference. Standard terms of reference are tailored to each evaluation.

The choice of consultants is made in the light of the specific characteristics of each evaluation, including the location of the interventions, familiarity with the sectors to be evaluated and particular linguistic knowledge.

The Evaluation Unit is responsible for providing the parties involved with information on the implementation of the evaluation exercise. To that end, it dispatches the terms of reference and the consultants CV to those parties (the relevant ECHO operational unit, other Commission departments where necessary, the partners responsible for implementing the intervention being evaluated, the ECHO correspondent and the Commission delegation).

The evaluation begins with a briefing at ECHOs headquarters involving the Evaluation Unit, the consultant and the country or regional desk of ECHO (and other Commission departments, where appropriate). The meeting allows the consultant to clarify any doubts there might be about the scope of his mission.

ECHOs Evaluation Unit ensures that the Commission delegation and the ECHO correspondent in the field do all they can to help the evaluation mission run smoothly. The consultant must refer to the Evaluation Unit any problems he has been unable to resolve on the spot.

At the end of the evaluation mission, the consultant must supply ECHO with a draft report which will be distributed within ECHO itself, to the appropriate Commission departments and to the partner.

ECHOs Evaluation Unit coordinates and chairs the debriefing meeting at which the evaluator presents the draft report to the parties involved in the evaluation. The partner can attend the debriefing meeting (at his own expense).

The Evaluation Unit distributes the final evaluation report to all the parties involved.

 

 

 

Evaluation questionnaire

 

Context :

> Analysis of the countrys political and socio-economic situation to the extent that this is needed to understand the emergency which the operation or global plan is seeking to cope with or the reasons for deciding to provide humanitarian aid.

> Needs assessment. In the field of humanitarian aid, needs assessment is virtually the same as identifying problems. A needs assessment is necessary to find out what the needs are and were, how many people are involved, and where the need is. It should be indicated who has performed the needs assessment a partner, the local authorities or local organisations (in case of medical interventions, hospitals or other healthcare facilities) and which method the evaluator used. Did the beneficiaries (individuals or institutions) take part in determining needs ? This is a vital question, particularly where disaster prevention and preparedness are concerned, because the involvement of the communities can be vital to the success of the intervention. In some situations (e.g. those requiring specialist equipment or involving rehabilitation) proof that the appropriate technical evaluation has been carried out and details of who performed it, is needed.

> Response capability. Any humanitarian aid intervention should be dictated by local response capabilities in the face of crisis. Attention therefore needs to be paid to the matter of what kind of assistance is provided by the local authorities and inhabitants for the victims, and the degree to which a country is vulnerable to and prepared for particular kinds of disaster (natural or man-made).

 

Relevance :

> Objectives. Objectives should be analysed to see if they have been/are valid in the light of the needs assessment, especially if operations began fairly late. Is the intervention a new one or a continuation of an existing one?

> Targeting. This covers the issues of how the beneficiaries (individuals or institutions such as hospitals or schools) are chosen ; whether the beneficiaries are a broad group or a smaller target group, how the groups were selected and what criteria were used ; also whether the choice of beneficiaries was justified in the light of the needs identified.

> Strategy. The issue is one of whether the strategies used were appropriate for the intervention in terms of duration, method of distribution, mode of transport, training activities and community involvement.

 

Coordination, coherence and complementarity :

The evaluator should find out if the intervention being evaluated is coherent, coordinated and complementary with the current (and short-term future) plans of other donors like the Member States, the US, the World Bank, UN agencies and other Commission departments.

Field coordination also needs to be examined: the relationship between the partners and the local authorities or the partners and the local contractors ; whether the partners are working in coordination with UN agencies or international NGOs active in the region; whether the partners have direct contact with the beneficiaries of the aid; what relations exist between the partners and the Commission delegation and/or the ECHO correspondent. Also in need of examination is the coordination of the intervention with the local authorities during implementation.

 

Effectiveness :

> Achievements. Examples would be distribution of x tonnes of maize ; establishment of x feeding centres .

> Results. These must be measured in terms of both quality and quantity. Was the aid suitable ? Did it go to the intended beneficiaries ? Was it of the intended quantity ? In the case of disaster-preparedness projects (but not only in such cases) it is particularly important to see what structures were set up (for example, improving local management capabilities and thus making an essential contribution to reducing vulnerability).

> Constraints. Any constraint beyond the partners control which has hampered fulfilment of the stated objectives must be reported (e.g. capabilities of the local authorities, partners or local subcontractors in charge of implementation; problems due to corruption, local mafias or security issues).

 

Cost-effectiveness analysis :

The cost effectiveness of an intervention is assessed by comparing that interventions cost with the cost of achieving the same objectives by other means. Because it is difficult to perform such an assessment for the whole of a humanitarian aid intervention, a realistic cost-effectiveness assessment tends to be confined to individual components such as the distribution of shipments of food or medicines, transport arrangements, etc.

 

Efficiency :

> Mobilisation. An assessment of the date of delivery to the point of arrival (forecast versus actual) and the length of time required for distribution to the beneficiaries; any delays in the implementation of these measures.

> Organisation of the operation in the field. How is the operation organised? Is there representation in the field (permanent or short-term)? Does the partner have the necessary communications and logistics equipment? Are its representatives in the field in a position to take immediate decisions in the light of circumstances, or do they have to await decisions from headquarters?

> Evaluation of storage and handling of goods; registration arrangements. What registration arrangements are made for taking delivery of the goods and managing stocks? Are suitable forms used? Are there provisions for security, and are they appropriate?

> Storage facilities. Where are the goods stored (warehouses, containers, etc)? To whom do the storage facilities belong? Are they suited to the quantity and type of goods involved? Have any losses occurred because of bad storage (too hot, too humid, in the open air, parasite infestation)? Were the storage arrangements cost-effective? How big are the reserves, and how long will they last?

> Suitability in terms of quality and quantity. Were the goods or services suited to the countrys diet and dietary habits? Did they match the contract specifications in terms of nature, content and quality ? Did they meet quality standards? Were use-by dates adhered to? Did the quantity delivered to intermediate and final destinations tally with the quantity originally supplied ? The suitability, quality and quantity of the goods must be checked with the beneficiaries.

> Packing. Were the products packed properly and in line with the specifications, or were there losses due to inadequate packing?

> Labelling and marking. Were the goods suitably labelled and marked?

> Purchasing and shipping. How were the goods or services bought and what did they cost? Where did they originate? Were contract-award procedures followed throughout? How was shipping arranged? What did it cost? Was transportation in accordance with the terms of the tender? Was the method of transport used the most viable or practical in the light of the circumstances?

> Distribution. How long did distribution take? On what dates did the goods enter the warehouses, go into intermediate distribution and go into final distribution? What was the actual duration compared to the expected duration?

> Distribution channels. What channels were used between the point of arrival and the point of final distribution? What stages were involved (e.g. hospital to patient)? How was registration of beneficiaries organised? Was it efficient? Were gender issues considered?

> Cost-sharing. Did the beneficiaries have to pay for the assistance or services received (e.g. for medical projects)? If they did, how did the institution involved record these sums?

> Monitoring. Did the partner set up a system for monitoring the operations, and was it efficient? At what stages did monitoring take place? In the case of medical projects, for instance, did the partner monitor distribution to the end of the chain, i.e. the patient? Did the partner monitor the distribution of the aid properly? Did the partner monitor the management of stocks properly?

> Self-evaluation. Did the partner set up a self-evaluation system? Did it measure efficiency and impact? If so, what impact indicators were used? Did the impact analysis take gender issues into account?

> Checks. Did the partner establish efficient checking mechanisms (e.g. for stocks or quality)? Are the financial control arrangements appropriate and transparent?

> Reporting. Is or was the partner adhering to the contracts reporting requirements (financial and narrative, for example)? Do the reports contain enough information on the progress made with implementing activities and on their results? Are they analytical?

 

Impact of ECHOs assistance :

An impact analysis makes it possible to determine if the operation or global plan has made any difference to the situation (compared with what it was before the intervention). It would find out, for example, if the intervention had lowered malnutrition, morbidity and mortality rates by increasing the calorie value of food or through vaccination. Because interventions can have an effect on local, regional or national markets, their effects on the supply and prices of goods produced locally or in the region should also be examined. Interventions may often have unexpected tangible or intangible advantages or disadvantages. Though they are difficult to quantify, they should still be analysed. Examples of advantages of this kind: improving the capabilities of partners and local authorities so that they can cope better with disasters (natural or man-made). Examples of disadvantages : deterioration of the local environment due to deforestation caused by chopping down trees for heating fuel ; discontent among the local population caused by a perception that refugees are better treated than themselves. In the case of large-scale and/or long-term food aid, impact analyses may consider measures to prevent long-term dependence on the aid. In the case of disaster-prevention projects, the most important impact yardstick is vulnerability, which the intervention should have reduced. This is difficult to gauge, because risk-reduction can be a longterm process. It should still be looked at, however, because it lies at the core of the project.

 

Gender issues :

Were gender issues taken suitably into account when the global plans and operations were being drawn up or implemented?

Security of relief workers :

Did the intervention have contingency plans for emergencies (evacuation, security or communications equipment for staff, etc.)? What steps were taken to ensure the safety of relief workers (both expatriate and local)? What communications facilities did they have? Were provisions made for emergency evacuation?

Viability :

The purpose of examining this matter is to determine if and for how long the effects of the intervention are going to last once outside aid has ceased. In the case of disaster preparedness, viability is obviously a vital consideration.

Viability is a particularly important aspect of projects intended to encourage self-sufficiency, in other words, to make displaced persons or refugees respond actively to their situation. Viability may also be a consideration with rehabilitation operations. Other viability issues also exist : for example, the establishment of coordination arrangements that continue to operate after the intervention has come to an end.

There is another purpose to considering viability : to see to what degree the ground has been prepared for humanitarian aid to give way to a development or cooperation phase.