Today, we are going to highlight a few different types of evaluations and three types of evaluation questions. Policy evaluation fits into a broad range of complementary methods that support evidence-based policy. While evaluations are periodic, objective assessments of a planned ongoing or completed project program or policy. Monitoring is a continuous process that tracks a program mid-stream, and uses data to inform program implementation and day-to-day management. Usually, monitoring tracks inputs, activities, and outputs, though occasionally it can include outcomes, such as progress towards national development goals. For example, monitoring a health insurance subsidy program would include how many people are accessing the website. Enrolling in the health insurance marketplace, accessing the subsidy, and so on. The evaluation however will focus on how many individuals get health insurance a new. So, in contrast to continuous monitoring, the evaluation is carried out at discrete points in time and often seeks an outside perspective from technical experts. The design method and cost varies substantially depending on the type of question the evaluation is trying to answer. Now, broadly speaking, evaluations can address three types of questions, descriptive, normative, and cause-and-effect. Lets take the first, descriptive evaluation questions seek to determine what is taking place. The results describe the processes, conditions, organizational relationships, and stakeholder views for an example. As an example, what are the characteristics of providers participating in ACOs? Are they primarily owned by physician practices or hospitals? Are they mostly nonprofit or for profit? These are descriptive questions, not asking what the ACO policy caused, but rather just about what is occurring. In contrast, our second type of question is the normative evaluation question. In which the evaluation compares what is taking place to what should be taking place. It assesses activities and whether or not targets are accomplished. Normative questions can apply to inputs, activities, and outputs. But the important attribute of normative questions is the use of the word should. Remember, the definition was that in normative evaluation questions, the evaluation compares what is taking place to what should be taking place. An example would be, are we as a society better off having an insurance individual mandate for all Americans to buy health insurance? This is not asking how the mandate affected the number of insured Americans, but rather whether societal welfare increased as a result of the mandate. It's hard to answer this without knowing how much we, the collective we, value everyone being insured versus having them insured but decreasing their take-home income to spend it on whatever they want to spend it on. We won't be diving further into these normative questions because of this value judgment valance that does not have to be a part of evaluation. See, the normative valuation compares what is taking place to what should be taking place. This means we are making that value judgement on what is best. Not just describing what happens. Depending on your point of view this can be either dangerous or near impossible, why? Because what should happen depends a lot on personal and societal values. Which can be difficult to precisely quantify or pin down. The third type of evaluation question is the cause-and-effect question. These evaluations examine outcomes and try to assess what difference the intervention makes in the outcomes. Taking our prior two examples, cause-and-effect questions would be; displacing providers in an ACO lead them to invest more in population management capabilities. Such as health information technology, or hiring care managers. Or, does passing an individual mandate lead to more Americans having health insurance. Generally speaking program and policy evaluations are of this cause-and-effect type. We want to go deeper than descriptive studies but stop short of value judgements and normative questions. Unlike general evaluations, which can answer many types of questions, policy evaluations are structured that one particular type of question. And that is the impact or causal effect of interest. The basic question incorporates this important causal dimension. We are interested only in the impact of the program. That is, it's effect on outcomes that the program itself directly causes. In other words, a policy impact evaluation looks at for the changes and outcomes that are directly attributable to the program or policy. See? We brought back those two pesky concepts of attribution and causality again. They will show up again and again and we will also have an entire session dedicated to causality, but more on that later. Today, we covered three types of questions. Descriptive, normative and cause-and-effect. And identify this third one cause and effect as the most relevant program in policy evaluation. Next time, we will keep forging ahead and more types of valuation to fill in the big picture. Thank you and see you then.
