Policy supporting business is rarely subjected to rigorous evaluation. As a result we have no reliable way of knowing if large sums of public money are being used effectively or wasted on initiatives which don’t work.
Previously there has been remarkably little use of rigorous experiment design in industrial and innovation policy. While large amounts of public money are spent supporting businesses around the world, nobody knows whether it’s having any real impact. In stark contrast with fields like medicine, new approaches are introduced without testing.
At a time when public resources are scarce it’s more important than ever that we find out what works and what doesn’t – and that principle should apply as much to business support as it does to programmes in healthcare or schools.
A Nesta report co-funded by the ESRC has used the ‘Creative Credits’ innovation scheme as a case to carry out a robust impact evaluation. In the scheme’s pilot in the Manchester City Region, 150 Creative Credits vouchers with a value of £4,000 were awarded to eligible SMEs to develop collaborative innovation projects with a creative business. The SMEs were required to contribute a further £1,000 of their own towards their project.
The researchers evaluated the Creative Credits scheme by using three key components:
- A randomised allocation of Creative Credits to firms in the ‘treatment’ group, comparing innovation and business performance with a ‘control’ group of firms made up of non–recipients.
- Collection of data over time, allowing the longer-term as well as short–term impacts to be assessed.
- A ‘mixed–methods’ approach combining qualitative in-depth interviews with quantitative survey techniques.
The evaluation showed that Creative Credits initially boosted innovation and sales growth in SMEs in the six months following completion of the scheme - but after a further six months these impacts were no longer statistically significant. This decrease in impact would have remained hidden with normal evaluation methods used by government, and indicates that randomised controlled trials should be used more widely in the evaluation of business support policies.
- Creative Credits created genuinely new relationships between SMEs and creative businesses. The award of a Creative Credit increased the likelihood by at least 84 per cent that firms would undertake their innovation project with a creative business they had not previously worked with.
- In the six months following completion of their creative projects, SMEs were significantly more likely than those that were not assigned Creative Credits to have introduced product and process innovations. The use of creative services also had a statistically significant positive effect on the sales growth of SMEs over the same period.
- However, 12 months after the completion of the project there was no longer a statistically significant difference between firms participating in the scheme and the control groups in the proportion of firms innovating - nor in sales growth.
- There was also no evidence that Creative Credits had had a permanent effect on the behaviour of SMEs. For example, those receiving Creative Credits appeared no more likely to work with creative service businesses in the longer term.
- The qualitative research suggests that for many SMEs where the short-term benefits were not sustained, the creative projects had been too ‘transactional’ in nature, and for others there had been communication difficulties with their creative partners.
Policy relevance and implications
This research brings together two recent developments in policymakers’ understanding of innovation. First, the recognition that the creative industries are an innovative sector that can stimulate innovation in other sectors too, through their supply chain linkages. Second, the argument that innovation policymakers should make use of controlled experimentation methods to trial and test new policy interventions.
- Innovation vouchers and other innovation support schemes should wherever possible be subjected to a randomised allocation as they are rolled out – ie businesses are randomly allocated to the participating group or to a non-participating control group. This will enable a sound assessment of the programme’s effectiveness before wider roll-out, and is a cheaper and simpler allocation method than subjective assessments.
- The use of randomised controlled trials is particularly important in situations with potentially severe selection bias – for instance, when the businesses that are selected to participate are already more innovative than the ‘average’ business the policymaker is targeting.
- Collection of data should also be carried out over time, in order to assess the longer-term effectiveness of different policy tools.
- The three-part evaluation approach in the Creative Credits project – randomised allocation of vouchers, longitudinal data collection, and the use of mixed methods - has proven to be effective, and should be used much more widely by the Government and other agencies in developing new innovation support policies.