I’m a great fan of the Integrated Evaluation Framework launched by AMEC in 2016, and fine tuned earlier this year.
Why? Well it’s an online tool that that is available to everybody; you don’t have to be a member of AMEC to access it. And it gets better in that it’s simple and free – now that’s my sort of resource!
I’m tempted to say that the devil is in the detail, but the devil has come nowhere near it. Instead, the strength of the framework is in the detail, specifically the guidance supplied in the excellent supporting documentation that underpins it. I’m a fan of the taxonomy developed for AMEC by Professor Jim Macnamara.
So, what I’ve done is trawled through the taxonomy and thrown in a few thoughts of my own to come up with nine steps to evaluation wisdom:
Do some proper situational and audience analysis before you start. Use PEST & SWOT analyses as well as stakeholder mapping. Don’t consign them to an appendix, take the revolutionary step of using them to inform your communications instead.
- Distinguish outputs, outtakes and outcomes.
- Focus on baselines (starting points) and benchmarks - comparative evaluation works well.
- Real women and real men use SMART objectives. Any other sort of objective is for wimps.
- Use the taxonomy to put concepts, metrics, methods and the like in the right place at the right time.
- Not all evaluation can measure impact, particularly in the short term. I like to separate intermediate outcomes (months) and long-term outcomes (years).
- Evaluating organisational, stakeholder and societal impacts requires an understanding of causation, correlations and econometrics.
- The taxonomy gives you the key rules of causation. For the rest, you will need to marry a statistician – your current partner won’t mind because when you retire and get back together again you’ll be rich.
- Do some proper research. And if you can’t, then piggyback on research others are doing (in the spirit of an omnibus survey). If that doesn’t work ask a passing expert who knows what is going on.
- Remember it’s all an iterative process. Tracking outputs will enable you to adjust your processes. Learn from outcome evaluation to improve the next campaign(s).
Finally, a tiny footnote about AMEC. It’s a rarity in changing its name but not its acronym. It is the international Association for the Measurement and Evaluation of Communications. It was the Association of Media Evaluation Companies. Not as trivial a point as it might seem: the change illustrates the positive developments in the communications evaluation industry over recent years.