Bedtime Experiments

I feel as if I have been a little tardy blogging recently. However, in addition to my own indolence I can blame sleep deprivation. Our little one year old has never been a great sleeper, but over a number of months had slipped into a fairly consistent and predictable routine in terms of how and when he would fall sleep and wake-up. That suddenly changed about two weeks ago. Settling him has become much harder, he’s starting waking earlier, and in addition has started being a lot more tearful and prone to getting upset during the day. Early on, his mother and I collapsed into the sofa late one evening and tried to work out between us what had caused this sudden change in behaviour. Between us we came up with six plausible explanations:

1. He was teething again (evidenced by copious drooling)
2. He had a cold (evidenced by a snotty nose)
3. Reaction to jabs (he had not long had a series of jabs after his first birthday, and we had been warned there may be some side effects)
4. Change in the weather (changes seemed to coincide with a distinctly autumnal seasonal shift including the temperature dropping and days noticeably shortening)
5. Developmental phase (the Touchpoints theory development by veteran paediatrician T. Berry Brazleton suggestions that babies development often occurs in spurts, and that these spurts can be accompanied by temporary regressions in other areas of development and behaviour. The little one had just started to walk and become more wilful in terms of indicating what he wanted to do, so it is possible that we were also seeing temporary regressions that were part of a significant development phase).
6. None of the above

One could of course add an additional cause, i.e. a combination of some/all of the above. We are keen to work out the cause(s) in the hope that we could respond appropriately, accepting that an appropriate response might just be to comfort him and wait for things to settle in their own time. Having said this, we did not want to do nothing if there was something we could do to make him feel better and more settled. How on earth could we work out what was causing his apparently sudden change in behaviour?

So why I am sharing this. First, I am really open to suggestions, and a little bit of crowd sourcing might just through up some useful insights. Second, our domestic situations feel analogous to many of the challenges that present themselves in the domains of health and social policy. The problems that we are trying to address often appear to have causes that are both multiple and unclear i.e. there is no single, clear cause and effect pathway.

How do we determine the exact causes of the problem, and what do we do if we can’t? Without being able to identify the causal pathway of the problem how do we decide how best to intervene?

Experiment vs. Context – finding the balance

Every so often one reads a paper, presentation or blog that manages, with brevity and clarity, to hit a bag full of nails on the head. Once the slight annoyance and feelings of inadequacy arising from one’s own inability to produce such a statement recede, it becomes possible to enjoy the fact that someone grappling with familiar issues to you has been able to make useful sense of them. In this case the document is a foreword (though I highly recommend the whole paper) by Naomi Eisenstadt to a recent British Psychological Association paper titled, Technique Is Not Enough: A framework for ensuring that evidence-based parenting programmes are socially inclusive.

The paper, and foreword, address the issue of parenting courses, but in that context, Naomi summarises a number of keys contemporary issues in the social policy domain. This includes how to balance the arguments advanced in favour of evidence-based practice based primarily on experimental studies (e.g. randomised control trials), against the arguments raised with regards the importance of context i.e. local needs, appropriateness, co-design and co-production of services, etc. In interest of some form of disclosure I must admit to being more drawn to the latter camp (perhaps as a result of being one of those humanities graduates that Ben Goldacre warns people about), so its good to hear a perspective that recognises those concerns, but also stresses the importance of trial-based evidence and the warns of the dangers than can arise from well-intended but poorly designed interventions.

Anyway,  I’ve attempted to summarise the key points of the Naomi’s foreword below, but I would highly encourage people to go to the source for an elegant and timely summary.

• Two camps exist – one emphasising randomised control trial evaluation and fidelity in implementation, the other emphasising the importance of context, including local knowledge and sensitivity to localised needs. The two positions need to be brought together.
• Common dilemma – those who most need support are those least likely to access it, and are also unlikely to be persuaded by logical arguments alone. Programmes will only engage families if they “speak to their needs and concerns.” (p5)
• Practical issues – its important to consider the practical barriers to participation e.g. transport, childcare, refreshments. (I particularly like Naomi’s suggestion that participation in parenting courses would rocket if venues offered free laundry facilities!)
• Barriers to scaling up – replication of RCT proven programmes requires that: 1) the programme is delivered with fidelity to the original design that was proven to work, 2) the delivery group needs to be similar to the original group. However, this present issues to repeating successful impact on a wider scale: 1) staff reluctance (for good and bad reasons) to exactly follow a programme designed elsewhere, and 2) difficulty in finding participants who closely resemble the original trial groups.
• Fidelity – is about adhering to key principals critical to effectiveness, rather than exact duplication (see also What we need in education is more integrity (and less fidelity) of implementation)
• Participation – the best programme will achieve little if no one takes part, so enabling participation is crucial (however, its also important to bear in mind that a bad programme that attracts participants could do harm and waste resources)

And finally in her own words:

“Whilst it is critically important to use programmes that are known to be effective, it is also crucial that such programmes and nuanced to local community circumstances.” (p6)

Collaboration, Networks and the Return of Silos

“I am in no doubt that effective support for children and families cannot be achieved by a single agency acting alone. It depends on a number of agencies working well together. It is a multi-disciplinary task.” (Laming, 2003)

One of the joys of the last year has been the arrival of a number of new trustees at the trust, and the opportunity this has afforded to me to work with some truly talented and good people. A drew the short straw, and ended up as my line manager, and in a short period of time I already feel that I have gained immensely from her experience and insights. A comes from a social care background, and before retiring was the manager of a children’s centre in one of the county’s most deprived wards. Having visited a number of the organisations funded by the trust, and read through several batches of quarterly reports, the same nagging worry comes up in our conversations i.e. the return of silo working.

Assuming our observations are correct the most likely culprit would seem to be the current round of funding cuts. Organisations, both voluntary and statutory, are facing severe financial pressures resulting in reduced hours, posts and capacity. Some of the organisations the Trust supports are fighting for their survival – on this basis sparing a member of staff to attend a multi-agency group meeting for the best part of a day seems neither practical nor a priority. In addition to reduced capacity there is the loss of staff knowledge and personal contacts, alongside the general churn of restructuring, that together can be corrosive of a more collaborative and networked way of working. The enthusiasm with which infrastructure services have been cut seems particularly myopic given the key role they have to play in facilitating networking, and seems to be part of a paradigm of ‘front line good, back office bad.’ From police services to children centres the drive appears to be to merge and share as many back office functions in pursuit of improved efficiency. Unfortunately, efficient is not the same as effective. Whilst there probably are many sensible savings to be made, it feels as if little consideration is being given to the impact that this is having on information sharing and collaborative working.

The dangers of silo working regarding the protection and safeguarding of children and vulnerable adults is clear. If the different organisations working with a child, family or vulnerable adult fail to share information, fail to pass on and follow-up concerns or exclude important voices, then the danger of significant harm is high. Central to Lord Laming’s recommendations were the needs for cross-boundary cooperation, information sharing and for different agencies to work together to support the needs of the child and their family.

However, beyond the immediate priority of safeguarding it seems to me that there is an even wider significance to Laming’s recommendations and the current return to silo working. That is the fact that many of the social problems faced by both voluntary and statutory organisations are complex, and therefore are beyond the scope of any one agency (or even a number acting in isolation) to address. Whether its school exclusions, social isolation, poverty, anti-social behaviour, etc. they are all manifestations of complex systems i.e. they have multiple and varied causes, are highly contextual, history-laden, evolving, dynamic and come about through the interaction of dozens (if not hundreds) of individual people with one another and their unique ecology. Not only are these problems difficult to solve, they are difficult to define. What does seem clear, and is of relevance to the re-emrgence of silo working, is that it is beyond anyone agency to understand and solve these social challenges. It seems to me that there is a very relevant connection here with say Dave Snowden’s work on complexity. Complex problems are beyond any one perspective or expertise domain to address, and it is an entirely sensible strategy when faced with such a problem to get as many relevant people around the table as possible in order to make sense of the situation and plan a response. In engaging with a complex system we are past the realms of ‘best practice’ i.e. there is no single ‘off-the-peg’ solution, however evidence-based it may be, that is guaranteed to be both appropriate and effective.

At the same time as picking up on the issues around silo working I have been reading a number of recent papers and articles (mainly from the US – H/T Joshua Sparrow) on the importance of collaboration and networking to achieve social impact. These include Stanford Social Innovation Review articles on collective impact, a lot of work by the Carnegie Foundation on Networked Improvement Communities and collected case studies of effective collaborations from across the US. The common theme throughout these studies would seem to fit coherently with the arguments loosely described above, namely that the “complex nature of most social problems belies the idea that any single program or organisation, however well-managed and funded, can single-handedly create lasting social change.” (Hanley-Brown et al. 2012) Below I have attempted a digest of what appears to be the key criteria for collaborations that effectively bring about social change:

• Develop a clear plan with a shared vision and agenda
• Shared measurement – allows partners to share information and hold one another accountable
• Involve all partners (inc. community members) from the start and throughout
• Work together to define the problem, solutions, targets, measurements and to produce the desired impact
• Data – use and share relevant data to set the agenda, evaluate and continuously improve over time
• Research-based – use both local and wider evidence to find out what works and respond
• Continuous communication – open sharing of data relevant to objectives across all partners
• Infrastructure – aka ‘backbone suport’ i.e. organisation dedicated to support the collaboration, and coordinate participation
• Commit to the long-haul – data on impact important to secure resources necessary for this commitment

It feels as it these lessons pose a number of challenges to philanthropic foundations. Part of the unique contribution and character of foundations and trust’s comes from their independence, particular passions and sometimes quirky way of working. However, what if this independence becomes isolation, and the failure to collaborate actually undermines a foundations efforts in achieving the social impact it desires? Should trusts simply provide the funds and stand back, or do they have a unique role to play in facilitating the formation of collective initiatives? Should foundations and other voluntary sector organisations be doing more to draw the government’s attention to the re-emergence and dangers of everyone retreating back into their silos?

;

;

;

;

Wellbeing and Evidence-based measurement

Updated 15th August 2012

The relatively quieter summer months are allowing for a little more reading. Over the last week or so, much of that reading has concerned wellbeing and the measurement thereof. Last week saw the publication of a number of reports on measuring national wellbeing released by the Office for National Statistics, and this week Measuring Wellbeing: A Handbook for Practitioners by the New Economics Foundation. I can recommend both, though I must concede I am still working through the ONS materials.

The NEF publication is well worth a read. Their dynamic model of wellbeing seems to make sense, and does well to recognise the importance of happiness at the same time as realising that there is far more to wellbeing than what can often be a passing emotional state. However, what particularly struck a chord for me was the report’s inclusion of evidence-based measurement tools, specifically in the form of validated and reliable questionnaires such as the ONS questions or the Short Warwick Edinburgh Mental Wellbeing Scale. It is also possible to compare any results that you collect through this scale with regional and national averages through NEF.

I have been thinking a lot about the question of evidence-based policy and practice of late (see previous post). One thought that struck me is that, from my limited perspective, is that much of the focus has been on defining and delineating what are evidence-based interventions. As previously noted, there are good reasons for this, but also a concern that this might limit innovation and adaptability in the commissioning of services (there is some discussion around whether the current evidence-based agenda will stifle or encourage innovative commissioning – but from my own provincial experiences I have seen more evidence of the former than the latter).

Whilst it is certainly not a question of either/or, I wonder whether it might be far more productive to concentrate the evidence-based discussion on promoting the use and adoption of evidence-based measurement tools, rather than attempts to produce prescriptive lists of evidence-based interventions. It seems to me promoting evidence-based measurement might bring a number of potential benefits:

1. Empowering and equipping community organisations to prove the value of what they do, and have created
2. Build the capacity of community organisations to demonstrate their impact
3. Enable groups to share their findings and promote their work
4. Expand, rather than restrict, the range of programmes that can be considered for funding

One of my concerns about the evidence-based agenda as it is currently manifesting (not necessarily the way it has to go, or is intended) is that is potentially very disempowering i.e. we the commissioners will tell you the providers what we think works, and what we are prepared therefore to fund. Not exactly equal partnerships and co-production.

I have taken the liberty of adding some more examples of validated and reliable wellbeing related measurement tools. Most are free to use, though as with the SWEMWBS may require permission:

• Rosenburg’s Self Esteem Scale (Rosenburg, 1989)
  
• Hospital Anxiety and Depression Scale (Zigmond, A. S., & Snaith, R.P. 1983)
• Everyday Feelings Questionnaire (Uher and Goodman, 2010)
• UCLA Loneliness Scale (Russell, 1996) (see http://www.tactileint.com/portfolio/uclalone.html)
• De Jong Gierveld Loneliness Scale (De Jong Gierveld, 1989)

Update 12th August 2012 (a couple more for good measure):

•  Satisfaction with Life Scale (Denier et al., 1985)
• Positive and Negative Affect Scale (PANAS) (Watson et al., 1998)

Evidence-based – some initial thoughts

One of the major current developments in both government and philanthropic thinking is the emphasis on evidence-based policy and practice. The argument goes that it is no longer enough to assume, or ‘to just know’ that your activities are making a difference to people. First, your activities (e.g. mentoring, parenting classes, sporting activities, etc.) need to be evidence-based. Ideally, this means that the programme you are delivering has been tried and tested before, and evidence collected to demonstrate that it consistently achieves the desired results. If your activity is new and innovative, then you still need to be able to point to evidence from similar programmes  that indicate your planned approach is sound and has a reasonable chance of working. Second, you need to be able to collect your own evidence that your activities are having the impact that you want them to have. An inability to demonstrate impact in a period of increasingly scarce funding is not a healthy characteristic.

There are different ways of collecting evidence, but the ‘gold standard’ for many is some form of randomised control trial (RCT). ‘Test, Learn, Adapt’ by Ben Goldacre et al. (of Bad Science fame) is a very interesting recent paper that both makes the case for evidence-based social policy, and gives a good ‘idiot’s guide’ to setting up one’s own RCT’s. Interesting examples of this approach that already appear to be having an influence on the funding of services include the list of 19 evidence-based early intervention projects compiled by Graham Allen or the National Academy for Parenting Research – Commissioning Toolkit. In some cases, especially with regards statutory funding, if your programme/intervention is not on the relevant list of evidence-based activities then it has a limited chance of receiving funding. These developments are not constrained to governments, and it already looks as if it will become increasingly common for independent trusts and foundations to emphasise the importance of evidence-based practice and RCTs (for example see Sean Stannard Stockton’s article at Tactical Philanthropy H/T Karl Wilding).

Personally, it feels as if there some advantages, and some dangers, in this push towards evidence-based practice. On the positive side, anything that increases the impact that both government and charities activity has on helping people in need has to be welcomed. When resources are limited, which they certainly are now, then it is vital to ensure that those resources are used as effectively as possible. Now for the ‘but’. It seems to me that there are some potential dangers.

• Innovation. A new approach to a addressing a social need will invariably have less evidence to draw on regarding its effectiveness, than one that is well established. There needs to be care that innovative, and potentially effective, new ideas are given opportunity to develop.
• Appropriateness. A colleague recently recalled a conversation with a commissioner who was waxing lyrical about the Family Nurse Partnership, an intervention aimed at young first-time mums, with a strong evidence base, having been evaluated for a number of years in the US. My colleague’s concern from the meeting was this – in terms of the needs presenting in the geographic area in question, young first time mums was not preeminent. In other words, there was a danger that funding would go towards an intervention because it had a strong evidence-base, not because it was what the community in question most needed.
• Complexity. Human beings are complex, and become even more so when rightly considered in the context of families and communities. First, this creates a challenges around RCTs – how do you conduct an experiment when they are so many different variables that could effect the outcome, and over which you have little control? Can you find a sample that is large enough to produce meaningful results, and ensure that baseline for each individual involved is comparable? If you can produce a controlled environment, will the results be replicable in environments that are different? What if the unit of intervention is not an individual, but an extended family or community? Where units of intervention are large and complex e.g. whole systems or communities, and the interventions themselves are necessarily multi-faceted, the limits of randomised control trials can become more evident.

It seems to me that RCT’s work best when you have a clear hypothesis, a large sample of genuinely comparable individuals, a controllable dosage (i.e. you can guarantee that every individual will have exactly the same intervention – or not, if they are in the control group) and you are looking to measure a single, specific outcome. This isn’t always possible and we need to look at mixed methods research and need to ensure that we are expanding the evidence universe, not reducing it, in order to try and capture everything that is happening in complex social interventions (more on this in a future blog post).

The issue of complexity is an important one as it reminds us that both charities and government are often operating in a fluid and dynamic environment, and if approaches  to evaluation are to reflect this they need to feed directly in organisation’s learning processes so that learning and programme development can be quick and responsive. RCTs tend towards summative, rather than formative, evaluation and as such “do not lend themselves to the real-time learning and fast program adaptations demanded by the complex and tumultuous environment in which nonprofits operate today.”

Listening to Troubled Families

I’ve just finished reading Louise Casey’s report for the Department for Communities and Local Government. It’s relatively succinct, and at its core are sixteen case studies of troubled families. It is a difficult and at times harrowing read. After the cases studies the report attempts to identify some of the main trends, and there are few surprises: shifting family make-up, abuse, dysfunctional relationships, mental health, drugs and alcohol, etc. The first, and perhaps most important theme to be identified is that of ‘Intergenerational Transmission.’ Problems such as abuse and neglect often seem to be passed on from generation to generation, with parents repeating the broken patterns of dysfunctional relationships and aggression that they experienced as children.

I think the following extract from the report’s conclusion bears repeating:

But what can be established, and perhaps the starkest message to take from these interviews, is the extent to which the problems of these families are linked and reinforcing. They accumulate across the life course, passed on from parents to their children across generations of the same family.

So this means that the traditional approach of services reaching individual family members, at crisis point or after, and trying to fix single issues such as ‘drug use’, ‘non-attendance at school’ or ‘domestic violence’ in these families is most often destined to fail. Their behaviours and problems can be properly understood only by looking at the full cycle – and the full family. This requires services who work with families to take the long view; of what happened to the parents as children and of what has happened to the children since birth.

I think that this is about right, but it raises for me the question of whether the current political system is actually able to deliver such a long-term and sustained vision. The government has pledged to turn around the lives of 120,000 of the most dysfunctional families by 2015. Notwithstanding the debate surrounding the figures, and definitions of troubled families, is it realistic to expect that entrenched, intergenerational problems can be resolved in a 2-3 years? Furthermore, with cuts hitting both local authorities and charities, especially infrastructure, where will the capacity come from to engage in this holistic and collaborative pattern of working?

My own sense is it is, and will be, community organisations and charities that will have the most most impact in this area. First, they are more likely to be rooted in the communities they serve, and have relationships with those families most in need – and it is ultimately positive relationships that transform people. Second, they are not bound by the  short-termism that drives both local and national political entities, with one eye constantly fixed on the media and the other on forth-coming elections. The obvious challenge, however, is funding. With cuts biting and the sector eating through reserves in order to survive, will there be the resources to support the vision?

Finally, something hinted at, but not made explicit in the report, is the need to recognise the importance of context and appropriateness. Every family is unique, and in the case of ‘troubled families’ not only are the problems they face multiple and compound, they are complex. In practical terms this means that it is highly unlikely that any one intervention, or even any particular suite of interventions is likely to work for every family. Any attempts to support vulnerable families will need to be bespoke, flexible and most importantly able to respond appropriately to the specific needs encountered.

Charity Law Review

Much excitement today re. Lord Hodgson’s Review of the 2006 Charities Act. It seems to be very wide-ranging,  with numerous recommendations, which in turn have generated quite a mixed response. It’s been interesting to note what aspects of the report people have picked up on, as much as their reactions. At times its felt a little like listening to people commenting on a Rosarch inkblot, with reactions perhaps telling your more about the commentator than the content. If you had been listening to Radio 5 today you could have been forgiven for thinking that this far-reaching review of the very nature of charities in England was actually a paper all about ‘chugging.’

NCVO’s response seemed to capture a common feeling of the report being a bit of a curate’s egg. Some of the finer points regarding joint reporting to the Charity Commission and Companies House will go down very well with dual-registered charities, and perhaps even slow the potential bolt for the CIO door, if such a route were to become available to existing organisations.

Through my narrow portal into the twitter-verse it looks as if the suggestion that “Charities who fall into the ‘large’ category [income over £1m] … should have the power to pay their trustees” (p42) is the recommendation that has generated the most animated response. Is this a pragmatic response to the challenge of attracting quality trustees, in a small number of cases to charities that employ thousands of staff and have multi-million pound turnovers? Or is it an unnecessary step, that would derive little benefit and run the risk of subverting the very principal of charity Trustee-ship? I am open to persuasion on this, but at present definitely lean towards the latter perspective. It feels to me as if this is an example of what Michael Sandel might describe as the move from a market economy to a market society, i.e. using a market mechanism (i.e. remuneration) to address an issue that is more to do with civic duty and public virtue.

3. “Theory of Change” – Reflection, Experiences, and Criticisms

In the last two posts I’ve written a little about what the ToC approach is, and how it works. In this third post on the ToC approach to strategic planning and evaluation I wanted to share some of my own experiences of implementing it in a work setting, reflecting upon what worked, what didn’t and the challenges and opportunities that were presented. To conclude I wanted to share some of the some of criticisms that exist of the approach.

Reflections and Experiences

In 2011 I led an action research investigation into the ToC approach in the context of the grant-making trust for whom I work. The trust in question was starting the process of developing its new strategic plan, to replace the 2007-12 plan that was starting to near the end of its life. What I wanted to know was whether the ToC approach could be used to improve the process of developing the new strategic plan, and whether it would help us as an organisation make a better job of both the process and the final product? I was also interested to see if any specific issues, good or bad, arose from the experience, especially from the perspective of being a relatively small organisation. There are six main points from this experience that I feel are worth sharing:

1. The ToC approach as defined in three key questions by Guthrie et al. (see previous posts) provided a useful structure for framing conversations, structuring reports, and reflecting back key strategic decisions. I found it a very useful way of structuring proposals and papers for Board meetings, both capturing decisions that had been made and identifying key questions that still needed to be answered. What was very striking when I cam to analyse the data collected from the Trustees (e.g. emails, research diaries, minutes, etc.) was the questions that were being asked e.g. “Could we do this…?”, “Will this work… ?”, “What skills do we need to achieve that..?”, etc. Compared to previous experiences, there was a striking level of very pertinent, and engaged, questioning.

2. The ToC emphasis on questioning assumptions around causality is important, but it was immediately apparent that to do this thoroughly would take time, and access to information and expertise that might not be immediately available. Looking at all of the relevant evidence, engaging stakeholders (inc. clients and experts), and carrying out an environmental analysis has proved to be very time consuming, and is still an ongoing process. For larger organisations, with several members of staff, this might not be too much of a challenge. However, as one of two members of staff (and the only one full-time) it was immediately apparent that this was not something that could be done quickly or easily.

3. Understanding, articulating and applying the ToC approach was made more difficult by the fact that the relevant literature contained a variety of definitions and explanations as to what the ToC approach is and how it should be implemented. This would also seem to have implications for the argument that the ToC approach could provide a sector wide common language.

4. The ToC approach places a clear emphasis on the primary importance of an organisation defining its ultimate goal. However, as we discovered, this is not always easy and is in itself a lengthy, complicated and potentially fraught process. There is nothing in the ToC approach that will tell you what it is you should ultimately be trying to achieve – you have to work that our for yourself.

5. The ToC approach might best be considered a ‘toolbox’ that contains a variety of models that may appeal to different foundations in different ways, depending on their capacity, philosophy and grant-making style. If, for example, you are a grant-making body that has clear goals, but is open to grant applicants pitching their ideas about how they will achieve those goals, then the more details aspects of the ToC approach (e.g. outcome chains) are probably not that relevant. In such a case it is the applicant who will be developing plans to show how their activities will lead to the desired change.

6. The ToC approach does not take place in a vacuum, but is affected by the environment in which it is applied. In this case the Trust experienced a challenging year, involving the sudden departure of a key Trustee, and a very complex resource management situation that had to be addressed. This led to delays. The ToC approach will not make internal governance issues, or external resource problems, go away.

Criticisms

As with any new, and popular, trend it is worth considering some of the potential criticisms and shortcomings that may exist.

First, there is the issue of the varying definitions regarding what the ToC approach is, and what it looks like when implemented. Whilst there is clearly a common thrust to the various explanations of the approach, there are also some significant variations, and as such there is a question as to whether this is one consistent and coherent model. Perhaps the most colourful criticism in this respect can be found in Albert Reusga’s paper Philanthropy’s Albatross: Debunking Theories of Change. There’s an interesting conference presentation (although sound quality is not great).

Second, again as hinted above, there is the question as to whether philanthropists and trusts should be developing Theories of Change, when really this is the domain of the organisations delivering the front-line services and interventions. This point is articulated in a very clear and succinct fashion by Sean Stannard-Stockton in part of an online discussion with Paul Brest. Using the business analogy of investors and entrepreneurs, Stannard-Stockton argues that foundations are investors. The entrepreneurs are the charities and nonprofits whose role it is to develop programmes and test their validity. In other words, it is the role of grantees, not foundations to develop and implement ToC.

Third, is the challenge of complexity, and this may prove to be the most significant challenge, not only to the ToC approach, but all logic model/programme theory approaches. Again, this is a point picked up by both Reusga and Stannard-Stockton, i.e. ToC work in static systems, not in complex and dynamic systems such as human society. The study of complexity and complex adaptive systems could have profound implications for the current practice of philanthropy.

Complex systems can be defined as having various characteristics including a large number of interacting elements and being dynamic compositions, where the whole is larger than sum of parts. It seems reasonable to suggest that the context in which many foundations are trying to effect change could be described as complex e.g. families, schools, communities, wider society, etc. Therefore the complexity theorist argument that this complexity “means that we cannot forecast or predict what will happen” would appear to have significant implications for the ToC approach, especially where trying to develop detailed causal outcome chains. Logic models and ToC outcomes chains are all predicated upon the idea that it is possible to map out a detailed cause and effect pathway for an intervention in a complex situation. This may not be possible.

2. “Theory of Change” – How?

Following on from the last post, I want to look in a little more detail at how the ToC approach can be used to develop an organisation’s strategic plan. As discussed, the starting point is to first work out what it is that you are trying to achieve or change? This is not always as easy as it sounds (and may well be a subject for a future blog post), but it is essential. Ideally, your ultimate goal should be clear and specific, otherwise it will be more difficult: a) to plan how to get there, b) monitor your progress, and c) work out if you have actually arrived at your intended destination.

So, having decided on a ultimate goal, it is the necessary to work how to get there. Initially, the conventional logic model is a very useful way of framing one’s thoughts:

Resources/Inputs – Activities – Outputs – Outcomes – Impact

A common approach in strategic planning is to work through the logic model backwards i.e. starting with your intended goal or impact, and then work through each of the steps backwards, in a logical sequence.

Where does the ToC approach come into the picture, and how does it add or help this process? Without being unduly critical it is helpful to consider one of the shortcomings of the logic model approach as described above, namely the causal connection between outputs and outcomes. For example, let’s imagine that we are trying to improve school readiness (i.e. young children arriving at school with the necessary social and practical skills necessary to enable them to participate fully in a mainstream educational setting – ability to play with others, basic reading skills, tie shoelaces, etc.) The belief is held that the best way to do this is to provide parenting classes over the course of 12 sessions, an hour per session, in a specific school catchment area. In logic model terms this would look something like this:

Outputs (Parenting Classes – 12x1hr for 100 parents) = Outcomes (Improved school readiness)

One could (and should) provide more details regarding both the classes and outcomes, but the key point is that there is a single causal arrow that connects the outputs to the intended outcomes. Run the parenting courses, so the model goes, and you will see significant improvements in the school readiness of the young children whose parents are participating. There is a belief that this approach (i.e. provision of parenting classes) is the best way for the organisation in question to achieve its ultimate outcomes (i.e. improved school readiness).

The first thing the ToC approach asks of this model is “Why?” “Why will parenting courses lead to this change in behaviour and development? Why do we believe this will work? What evidence to we have that this approach is needed, appropriate and likely to be effective? In this respect the ToC approach can be seen as linking in quite strongly with the current moves towards a more evidence-based approach to social policy and intervention.

The second thing that the ToC approach asks of this model is whether a single causal connection is an adequate explanation what needs to happen for the desired change to occur. One does not have to be a veteran community worker to suggest that improving school readiness, particularly in a community that faces a wide range of challenges, is unlikely to be simple. Effecting change is actually going to be a complex and demanding task that will probably require a range of efforts, and will probably involve a journey with succession of small, incremental steps.

In considering the actually causal connection between your organisation’s activities and the desired change it is useful to consider the following:

• Evidence – what evidence is there the planned intervention will lead to the desired outcome?

The evidence-based approach is very much a ‘hot-topic’ in social policy and interventions at the moment. Resources are growing in this respect, and using the example of early intervention and children’s work one could look at sites such as C4EO, The WAVE Trust, and Zero to Three and the recent reports produced by Graham Allen. However, experience can also include your own organisation’s data and experiences, and evidence collected from the communities with whom you work.

• Single or multiple steps – will the intervention lead directly to the desired outcome, or are there a number of “micro steps” along the way? Is there a chain of connected outcomes?  

The “outcomes-chain” is one of the most distinctive features of the ToC approach. Rather than a single connection between output and outcome (i.e. what you do and the results this generates), the “outcomes-chain” depicts a series of sequentially connected outcomes that lead, step-by-step, to the programmes ultimate outcome. Using the example above, it may be that:

Parenting Classes  = Improved school readiness

Might looks a bit more like this:

Interview parents, nursery and school staff so that specific support needs can be identified so that appropriate parenting support can be provided so that parents are able to engage with accessible and appropriate activities so that their parenting skills improve and so that  additional needs can be identified so that parents can be signposted on to additional services so that comprehensive support is provided to the whole family so that children’s full developmental needs can be met (including any specific educational needs) so that they readiness for school is improved

• Linear or multifaceted – is the outcomes chain a simple linear process or is it multifaceted

Do separate outcomes chains need to run in parallel in order to combine towards the final, ultimate outcome? One way to think about this is to talk about what is necessary for change and what is sufficient? A necessary condition is one that has to be met for change to occur, but it is not enough to bring about change itself e.g. it may be necessary for a parent to engage in the parenting classes for their skills and awareness to improve, but that is not enough by itself to ensure that their child’s school readiness improves. It may be in addition to the classes that they also need to access additional support to help meet their child’s specific leaning needs. However, some conditions (or combination thereof) may be sufficient, that is they are enough to effect change without anything else having to happen. In the case of complex social interventions, it is often the case that a number of conditions have to be met in order for change to occur, hence the need of comprehensive and collaborative (i.e. inter-agency) support.

• Internal and External factors – does the outcomes chain take into account the influence of internal factors (peoples, systems, and resources) and external factors (e.g. changes in government policy, economy, or population)

Some times referred to as “programme factors” and “non-programme factors”. Programme/Internal factors are those things that you can control e.g. the number and location of parenting classes. Non-programme/External factors are this things that you cannot control, but could influence the progress of your programme e.g. closure of a school or community centre, a recession, significant population change, etc. Whilst it is not possible to control external factors, it is important to be aware of their presence and the impact that they have on your programme. Partly this is about being able to respond to changes in your operating environment, but it is also about recognising that any changes you observe (good or bad) may not be entirely due to your actions.

• Plausibility and coherency – ultimately any plan that details a causal link between an intervention and an outcome needs to be scrutinised, ideally by those who have knowledge relevant to the intended change (e.g. practitioners, academic researchers and intended beneficiaries).

 Any ToC or logic model will have to be implemented in a real-world context. It may look great on paper, but those stakeholders with a personal and profound understanding of this issues being addressed will need to be able to input into the process in order to ensure that the proposed plan has any chance of working.

I can highly recommend “Purposeful Program Theory” by Funnell and Rogers for a comprehensive exploration of these issues (though its not exactly a light bedtime read!). For a more succinct introduction, but still covering “outcomes-chains” etc. Brian Lamb’s paper for NCVO is a good place to start.

1. “Theory of Change” – What?

The “Theory of Change” approach can be defined as a particular approach to strategic planning and evaluation that has been developed by and for organisations intent on effecting social change. There are a number of different specific definitions as to what exactly the ToC approach is and consists of (an issue that I’ll address in the third post in this series), but perhaps the most useful one I have found to both understand and use is this:

“No matter how it is defined, at its heart, a theory of change lays out what specific changes the group wants to see in the world, and how and why a group expects its actions to lead to those changes. The important elements are defining very clearly the end goals of the program and hypothesizing the strategies the group is using to move toward achieving those goals.’”(Guthrie et al. 2005 – emphasis by Lamb, 2011, p6) 

As with a number of other approaches (e.g. logic models) the ToC approach emphasises the primary importance of having a clear picture of what it is that you are trying to achieve i.e. “What?”. This could be reducing poverty, increasing opportunities to participate in sport, or improving support for people with a specific health need – without a clear vision of where you are going, its very hard to plan how to get there and to know when you have (or have not) arrived. The next step is to answer the “How?” question. Again, the ToC approach is very similar to the more familiar Logic Model and other programme theory approaches in that it stresses the importance of having a clear understanding of how the activities that you undertake will lead to the changes and results that you want to achieve. In strategic planing terms it is not uncommon to do a ‘reverse logic model’ i.e. take the standard model (inputs-activities-ouputs-outcomes-impact) and start at the end, with the desired results and then work out sequentially backwards what you need to do to get there.

So far, so familiar. So what does the ToC approach bring to the table. There are probably features of the ToC approach that make it distinctive and these come from the third question in the definition above – “Why?’ What the ToC approach does is challenge philanthropic organisations to look at their plans, specifically the activities they outline, and ask if they really will lead to the results and changes described. We will look at this in more detail in the next post, but this type of self-questioning leads quickly in to contemporary discussions such as the role of evidence and importance of involving experts, including the people being targeted, in the design of services.

One of the key feature of the ToC is the recognition that bringing about lasting social change is rarely straightforward or simple. One of the implied shortcomings of the Logic Model is that there is often a single arrow that links the activity with the desired result. Again, more on this in the next post, but the ToC approach would see this as too simplistic and in reality a whole series of activities and small, sequential changes might need to take place in order for the desired change to occur.

Put a little more formally the ToC appears to have al treat two distinctive features i.e. the scrutiny and validation of linkages and the importance of political and social context:

The scrutiny and validation of linkages – The ToC approach seeks to make explicit the underlying assumptions how certain activities will affect certain changes. Having identified those assumptions it is then possible to draw on research-based evidence and expert/stakeholder input to assess whether they are valid.  

The importance of political and social context: – The important element here is the recognition that change happens in social and political contexts that are not of an organisation’s making. It is possible to have a strategic plan/logic model that is entirely coherent, but fails because it did not take into consideration the influence of factors beyond the organisation’s control.