In the notice of funding opportunity (NOFO) for the CDC’s OD2A initiative, it states that “While the opioid overdose epidemic worsens in scope and magnitude, it is also becoming more complex.” The NOFO also states that “The complex and changing nature of the opioid overdose epidemic highlights the need for an interdisciplinary, comprehensive, and cohesive public health approach.”
As evaluators funded by the OD2A cooperative agreement, we are faced with the daunting task of providing credible evidence and justified recommendations for the effective prevention of, and response to, a complex and ever shifting problem that requires multifaceted and coordinated solutions.
And while the conventional approach to evaluation of public health problems has served society well, it is becoming increasingly clear that our tried-and-true methodologies are insufficient to the task of understanding the scope, direction, and contours of this epidemic and of driving well-informed and necessarily adaptive programmatic decision-making.
This is why I am encouraging our OD2A Eval CoP to consider the alternative frame for evaluation that is discussed in this new book by Thomas Schwandt and Emily Gates: Evaluating and Valuing in Social Research. I believe that the emerging orientation to complex (AKA wicked) social problems espoused by the authors dovetails nicely with the evaluation profile that was recently introduced by the CDC’s Division of Overdose Prevention. Both the alternative frame for evaluation and the evaluation profile have been developed to address wicked problems like the ongoing overdose crisis.
To get a conversation going in our community about the book, I will highlight a couple of aspects of the alternative frame for evaluation discussed in the book that seem pertinent to the work that we are doing.
Systems Thinking and Complexity Science
One aspect of the alternative frame for evaluation is the use of systems thinking and complexity science (STCS). STCS is problem-focused instead of solution-focused, since it is assumed that wicked social problems stem from values that are fluid, and from purposes that are unstable and pluralistic, and require a multidisciplinary approach to make sense of them.
With the conventional frame of evaluation, we assume that problems function independently, problems remain relatively stable, the right solution will reduce or eliminate the problem, and that evaluation is meant to appraise solutions to already defined problems. In other words, we believe that we have the problem figured out and that we just need to apply the right solution (often referred to as evidence-based practice).
The alternative frame of evaluation, on the other hand, recognizes that wicked problems are interconnected and influence other problems. And since every problem definition implies different solution options, and since improvements associated with solutions require boundaries of the evaluation of those solutions, boundary judgment and critique are necessary.
When applying boundary judgment, we need to appreciate how different stakeholders frame problems, strategies, and outcomes, and how these different perspectives need to engage one another in a meaningful way. And, since setting indicators for evaluations are also boundary decisions, and are therefore value judgments, broad and intentionally inclusive stakeholder participation is crucial to establishing useful and feasible indicators.
Co-production – emphasizing deliberation over analysis.
Another aspect of the alternative frame for evaluation is the use of an iterative and collaborative process involving multiple types of expertise, actors, and knowledge to answer how did we do and what should we do now. In the book, this is called co-production. Co-production emphasizes deliberation over analysis. Analysis uses rigorous methods to arrive at answers to factual questions. With deliberation, people ponder and confer on matters of mutual interest to negotiate and persuade each other and may include both consensual communication and adversarial communication.
In the process of co-production, there is a thoughtful, practical doing called praxis that is used to facilitate social learning. This social learning is guided by the participants’ impressions, the social values that act as normative guides, a political strategy that is chosen to obtain desired results, and social actions which consist of activities to implement the political strategies. When the focus is social learning, the role of the evaluator shifts from solely providing information to facilitating change and adaptive management.
In a world beset by ‘wicked’ problems.
These are just a couple of the concepts in the book that reconceptualize the role of evaluation and evaluators in a world beset by ‘wicked’ problems. The overall message is that evaluators must take on a bolder agenda to contribute to change, not just providers of evidence but also stimulating deliberation over the evidence by a broader range of citizens.
We need to act less like external judges and more like engaged thoughtful partners to help groups make sense of pervasive social issues like the overdose crisis and to steer a path of action through these difficult situations.
Leave a Reply