Mar 11 2015 Evaluating cross sector initiatives through a lens of complexity

Complex social problems invite intersector solutions — those that rely on the resources of government, business, and non-profit sectors. Because collaborations involve complex inputs (multi-stakeholder partners and a host of different approaches and resources) the outcomes of intersector projects can be unpredictable with many variables interacting in unknown ways.

“How can the practice of evaluation evolve to better reflect the complexity of social problems and their solutions today?” This was question tackled in an FSG webinar we tuned into today titled “What Makes Evaluating Complexity Different?” The webinar brought FSG’s Hallie Preskill, Srik Gopal, and Katelyn Mack in conversation with Cris Kutzli, Program Director at Grand Rapids Community Foundation, and Mona Jhawar, Evaluation Manager at The California Endowment.

A few weeks ago, we wrote a blog post about the FSG report that served as the basis for this webinar, entitled “Evaluating Complexity: Propositions for Improving Practice.” Principles FSG highlighted in this report include:

• Complex systems are never static.
• Events in one part of the system have ripple effects in others.
• Context matters.
• Understanding relationships between entities is equally if not more important than understanding the entities themselves.

Discussion today used the report as a jumping point to hone in on the ways in which practitioners, researchers, and evaluators need to be more skilled at anticipating, seeing, and responding to surprises when evaluating initiatives.

Cris Kutzli discussed a partnership among Grand Rapids Public Schools, Grand Rapids Community Foundation, and FSG. The program, Challenge Scholars, offers a scholarship upon graduation and additional support to students and their families starting when the student is in sixth grade. She explained that, to evaluate the program, it was important to understand the five primary guiding questions of the program:

• How is a college and career culture emerging within the schools and the broader community?
• How are parents and families being engaged as part of the CS?
• How are students becoming more college and career ready?
• What does exemplary teaching and learning look like in a colleges and career culture?
• How are the community, businesses, and higher education responding to CS?

Both the design and implementation stages of an evaluation need to be adaptable and iterative, Kutzli said. In order to do this, the evaluation plan was co-created with the working group. “Co-creation and iteration is not something you do once,” she said. These activities form the basis of the design and implementation of the initiative. “Co-create, iterate, rinse, repeat” creates buy-in early in the initiative and firmly establishes the evaluation team as learning partners. This helps to build trust and enhances the team’s capacity to work better together.

Mona Jhawar described a program called Building Healthy Communities, which supports the development of communities where kids and youth are healthy, safe, and ready to learn. The organization partnered with FSG to conduct a strategic review to reflect on their work, aiming to use the findings to strengthen the initiative and improve their results overall. She said that, when evaluating, it is important to pay attention to context and respond to changes as they occur. Most importantly, evaluators should understand the characteristics of the people and setting, the economic environment, the way that institutions interact with one another, and the political dynamic of a place.

What FSG calls a “complexity lens” implies not a single methodology, but instead it is a lens through which evaluators should employ these nine propositions:

• Design and implement evaluations to be adaptive, flexible, and iterative.
• Seek to understand and describe the whole system, including components and connections.
• Support the capacity of the system to learn by strengthening feedback loops and improving access to information.
• Pay particular attention to context and be responsive to changes as they occur.
• Look for effective principles of practice in action rather than assessing adherence to a predetermined set of activities.
• Identify points of energy and influence, as well as ways in which momentum and power flow with the system.
• Focus on the nature of relationships and interdependencies within the system.
• Explain the non-linear and multi-directional relationships between the initiatives and its intended and unintended outcomes.
• Watch for patterns, both one-off and repeating, at different levels of the system.

Using a “complexity lens” when evaluating intersector projects can help us embrace the complexity of cross sector collaborations rather erase it.