Explain two reasons why high quality impact evaluations incl

Solution

1)

Sample selection problems could arise if factors other than random assignment influence program allocation. For example, parents may move their children from a class (or a school) without the program to a school with the program. Conversely, individuals allocated to a treatment group may not receive the treatment (for example, because they decide not to take up the program). Even if randomized methods have been employed and the intended allocation of the program was random, the actual allocation may not be. This problem can be addressed through “intention to treat (ITT)” methods or by using random assignment as an instrument of variables for actual assignment. Although the initial assignment does not guarantee in this case that someone is actually either in the program or in the comparison group, in most cases it is at least more likely that someone is in the program group if he or she was initially allocated to it. The researcher can thus compare outcomes in the initially assigned group and scale up the difference by dividing it by the difference in the probability of receiving the treatment in those two groups to obtain the local average treatment effect estimate. Methods such as ITT estimates allow selection problems to be addressed fairly easily in the context of randomized evaluations, but it is often much more difficult to make these corrections in the case of a retrospective analysis.

2)

A second issue affecting both randomized and non-randomized evaluations is differential attrition in the treatment and the comparison groups: those who participate in the program may be less likely to move or otherwise drop out of the sample that those who do not. Statistical techniques can be used to bound the potential bias, but the ideal is to try to limit attrition as much as possible.