Experiment vs. Context – finding the balance

Every so often one reads a paper, presentation or blog that manages, with brevity and clarity, to hit a bag full of nails on the head. Once the slight annoyance and feelings of inadequacy arising from one’s own inability to produce such a statement recede, it becomes possible to enjoy the fact that someone grappling with familiar issues to you has been able to make useful sense of them. In this case the document is a foreword (though I highly recommend the whole paper) by Naomi Eisenstadt to a recent British Psychological Association paper titled, Technique Is Not Enough: A framework for ensuring that evidence-based parenting programmes are socially inclusive.

The paper, and foreword, address the issue of parenting courses, but in that context, Naomi summarises a number of keys contemporary issues in the social policy domain. This includes how to balance the arguments advanced in favour of evidence-based practice based primarily on experimental studies (e.g. randomised control trials), against the arguments raised with regards the importance of context i.e. local needs, appropriateness, co-design and co-production of services, etc. In interest of some form of disclosure I must admit to being more drawn to the latter camp (perhaps as a result of being one of those humanities graduates that Ben Goldacre warns people about), so its good to hear a perspective that recognises those concerns, but also stresses the importance of trial-based evidence and the warns of the dangers than can arise from well-intended but poorly designed interventions.

Anyway,  I’ve attempted to summarise the key points of the Naomi’s foreword below, but I would highly encourage people to go to the source for an elegant and timely summary.

• Two camps exist – one emphasising randomised control trial evaluation and fidelity in implementation, the other emphasising the importance of context, including local knowledge and sensitivity to localised needs. The two positions need to be brought together.
• Common dilemma – those who most need support are those least likely to access it, and are also unlikely to be persuaded by logical arguments alone. Programmes will only engage families if they “speak to their needs and concerns.” (p5)
• Practical issues – its important to consider the practical barriers to participation e.g. transport, childcare, refreshments. (I particularly like Naomi’s suggestion that participation in parenting courses would rocket if venues offered free laundry facilities!)
• Barriers to scaling up – replication of RCT proven programmes requires that: 1) the programme is delivered with fidelity to the original design that was proven to work, 2) the delivery group needs to be similar to the original group. However, this present issues to repeating successful impact on a wider scale: 1) staff reluctance (for good and bad reasons) to exactly follow a programme designed elsewhere, and 2) difficulty in finding participants who closely resemble the original trial groups.
• Fidelity – is about adhering to key principals critical to effectiveness, rather than exact duplication (see also What we need in education is more integrity (and less fidelity) of implementation)
• Participation – the best programme will achieve little if no one takes part, so enabling participation is crucial (however, its also important to bear in mind that a bad programme that attracts participants could do harm and waste resources)

And finally in her own words:

“Whilst it is critically important to use programmes that are known to be effective, it is also crucial that such programmes and nuanced to local community circumstances.” (p6)

Collaboration, Networks and the Return of Silos

“I am in no doubt that effective support for children and families cannot be achieved by a single agency acting alone. It depends on a number of agencies working well together. It is a multi-disciplinary task.” (Laming, 2003)

One of the joys of the last year has been the arrival of a number of new trustees at the trust, and the opportunity this has afforded to me to work with some truly talented and good people. A drew the short straw, and ended up as my line manager, and in a short period of time I already feel that I have gained immensely from her experience and insights. A comes from a social care background, and before retiring was the manager of a children’s centre in one of the county’s most deprived wards. Having visited a number of the organisations funded by the trust, and read through several batches of quarterly reports, the same nagging worry comes up in our conversations i.e. the return of silo working.

Assuming our observations are correct the most likely culprit would seem to be the current round of funding cuts. Organisations, both voluntary and statutory, are facing severe financial pressures resulting in reduced hours, posts and capacity. Some of the organisations the Trust supports are fighting for their survival – on this basis sparing a member of staff to attend a multi-agency group meeting for the best part of a day seems neither practical nor a priority. In addition to reduced capacity there is the loss of staff knowledge and personal contacts, alongside the general churn of restructuring, that together can be corrosive of a more collaborative and networked way of working. The enthusiasm with which infrastructure services have been cut seems particularly myopic given the key role they have to play in facilitating networking, and seems to be part of a paradigm of ‘front line good, back office bad.’ From police services to children centres the drive appears to be to merge and share as many back office functions in pursuit of improved efficiency. Unfortunately, efficient is not the same as effective. Whilst there probably are many sensible savings to be made, it feels as if little consideration is being given to the impact that this is having on information sharing and collaborative working.

The dangers of silo working regarding the protection and safeguarding of children and vulnerable adults is clear. If the different organisations working with a child, family or vulnerable adult fail to share information, fail to pass on and follow-up concerns or exclude important voices, then the danger of significant harm is high. Central to Lord Laming’s recommendations were the needs for cross-boundary cooperation, information sharing and for different agencies to work together to support the needs of the child and their family.

However, beyond the immediate priority of safeguarding it seems to me that there is an even wider significance to Laming’s recommendations and the current return to silo working. That is the fact that many of the social problems faced by both voluntary and statutory organisations are complex, and therefore are beyond the scope of any one agency (or even a number acting in isolation) to address. Whether its school exclusions, social isolation, poverty, anti-social behaviour, etc. they are all manifestations of complex systems i.e. they have multiple and varied causes, are highly contextual, history-laden, evolving, dynamic and come about through the interaction of dozens (if not hundreds) of individual people with one another and their unique ecology. Not only are these problems difficult to solve, they are difficult to define. What does seem clear, and is of relevance to the re-emrgence of silo working, is that it is beyond anyone agency to understand and solve these social challenges. It seems to me that there is a very relevant connection here with say Dave Snowden’s work on complexity. Complex problems are beyond any one perspective or expertise domain to address, and it is an entirely sensible strategy when faced with such a problem to get as many relevant people around the table as possible in order to make sense of the situation and plan a response. In engaging with a complex system we are past the realms of ‘best practice’ i.e. there is no single ‘off-the-peg’ solution, however evidence-based it may be, that is guaranteed to be both appropriate and effective.

At the same time as picking up on the issues around silo working I have been reading a number of recent papers and articles (mainly from the US – H/T Joshua Sparrow) on the importance of collaboration and networking to achieve social impact. These include Stanford Social Innovation Review articles on collective impact, a lot of work by the Carnegie Foundation on Networked Improvement Communities and collected case studies of effective collaborations from across the US. The common theme throughout these studies would seem to fit coherently with the arguments loosely described above, namely that the “complex nature of most social problems belies the idea that any single program or organisation, however well-managed and funded, can single-handedly create lasting social change.” (Hanley-Brown et al. 2012) Below I have attempted a digest of what appears to be the key criteria for collaborations that effectively bring about social change:

• Develop a clear plan with a shared vision and agenda
• Shared measurement – allows partners to share information and hold one another accountable
• Involve all partners (inc. community members) from the start and throughout
• Work together to define the problem, solutions, targets, measurements and to produce the desired impact
• Data – use and share relevant data to set the agenda, evaluate and continuously improve over time
• Research-based – use both local and wider evidence to find out what works and respond
• Continuous communication – open sharing of data relevant to objectives across all partners
• Infrastructure – aka ‘backbone suport’ i.e. organisation dedicated to support the collaboration, and coordinate participation
• Commit to the long-haul – data on impact important to secure resources necessary for this commitment

It feels as it these lessons pose a number of challenges to philanthropic foundations. Part of the unique contribution and character of foundations and trust’s comes from their independence, particular passions and sometimes quirky way of working. However, what if this independence becomes isolation, and the failure to collaborate actually undermines a foundations efforts in achieving the social impact it desires? Should trusts simply provide the funds and stand back, or do they have a unique role to play in facilitating the formation of collective initiatives? Should foundations and other voluntary sector organisations be doing more to draw the government’s attention to the re-emergence and dangers of everyone retreating back into their silos?

;

;

;

;