Home / Summaries / Evaluation A Systematic Approach / control-outcome-selection

Assessing Program Impact: Alternative Designs

18 important questions on Assessing Program Impact: Alternative Designs

Nonequivalent comparison design

a quasi experimental design in which intervention and control groups are constructed through some means other than random assignment. equivalence on outcome, absent program exposure, cannot necessarily be assumed

Intention of evaluator when conducting impact assessment

a fair and accurate estimate of program's actual effects, hence RCT when feasible

Bias comes into picture when:

either measurement of the outcome with origami exposure or the estimate of what the outcome would have been without program exposure is higher or lower than the corresponding true value.

When the level of either of these outcomes is misrepresented in the evaluation data, the estimate of the program effect, in tun, will be smaller or larger than the actual program effect (it will be biased)

How can you avoid bias in measuring the observed outcome for targets exposed to an intervention

using measures that are valid and responsive to the full range of outcome levels likely to appear among the targets

Before-after measures almost always produce biased estimates of program effects because:

natural changes over time in many aspects of human behaviour

Assumption of program effect regarding outcome measures

If neither intervention or control group received the intervention, their mean outcome scores would be the same over the pertinent pre-post time interval

Attrition from _____ degrades design.

outcome measurement

3 categories of experiences and events that may produce bias in impact assessments

1) secular trends
2) interferring events
3) Maturation

Techniques to construct control groups

1) Matching
2) equating groups by statistical procedures
3) reflexive controls

How do you know which characteristics are essential to match?

through prior knowledge and theoretical understanding of the social process in question
- identify variables that are potentially related to selection process that divide targets into program participants (if you cannot match it is important to still identify them to incorporate them into data analysis and perhaps statistically adjust for remaining selection bias)

2 types of matching procedures

Individual matching (usually preferable to aggregate matching, but may be difficult to execute and can result in drastic loss of cases)
aggregate matching

There is ____ that statistical control of variables would completely remove selection bias because

no assurance
influential differences between the interventions and control groups might still remain

2 different types of control variables in nonequivalent comparison group designs

- initial characteristics of group members that are related to the outcome variable (if intervention has no effect, these characteristics 'predict' the outcome)
- control variable that relates directly to selection bias, and has to do with selection of individuals into the intervention vs control group. With this control variable you could fully account for the characteristics that caused an individual to be selected for one group or the other, statistically control on these variables and perfectly offset the selection bias.

Objective of multivariate analysis of data from nonequivalent comparison groups

is to construct a statistical model that predicts each individual's value on the outcome variable from the control variables measured at the beginning of the study

Multivariate analysis tries to

establish whether or not receiving the intervention in itself is a significant predictor of outcome when predictive relationship of control variables has already been taken into account.

Propensity score analysis

composite selection variable resulting from the stage one analysis can, alternatively, be used as a matching variable to create subsets of intervention and control groups that are equivalent in that composite selection variable

The advantage of making fewer statistical assumptions that the two-stage multiple regression approach.

Disadvantage: the ability to statistically adjust away selection bias is heavily dependent on identification of the variables related to selection into groups and inclusion of them in the statistical model that generates the propensity scores.

Often cutting-point designs have not been applied, why?

- not all programs have definite and precise rules for eligibility or are willing to adopt such rules for purposes of impact assessment
- they are not well understood

When are pre-post tests most appropriate?

for short-term assessments for programs attempting to affect conditions that are unlikely to change much on their own

The question on the page originate from the summary of the following study material:

Evaluation A Systematic Approach | 9780761908944 | Peter H Rossi, et al

View summary

A unique study and practice tool
Never study anything twice again
Get the grades you hope for
100% sure, 100% understanding

Remember faster, study better. Scientifically proven.

Assessing Program Impact: Alternative Designs

18 important questions on Assessing Program Impact: Alternative Designs

Nonequivalent comparison design

Intention of evaluator when conducting impact assessment

Bias comes into picture when:

How can you avoid bias in measuring the observed outcome for targets exposed to an intervention

Before-after measures almost always produce biased estimates of program effects because:

Assumption of program effect regarding outcome measures

Attrition from _____ degrades design.

3 categories of experiences and events that may produce bias in impact assessments

Techniques to construct control groups

How do you know which characteristics are essential to match?

2 types of matching procedures

There is ____ that statistical control of variables would completely remove selection bias because

2 different types of control variables in nonequivalent comparison group designs

Objective of multivariate analysis of data from nonequivalent comparison groups

Multivariate analysis tries to

Propensity score analysis

Often cutting-point designs have not been applied, why?

When are pre-post tests most appropriate?

Summaries related to Expressing and assessing program theory

Evaluation A Systematic Approach

Realistic Evaluation

Handbook of bereavement research: consequenc…

Western civilization : beyond boundaries

Anthropological theory : an introductory his…

Biopsychology

PMP - project management professional study …

Biology: A Global Approach, Global Edition

Principles of Marketing

Britain - for Learners of English

Business vocabulary in use intermediate

Organizational behaviour