The Innovation Fund

Applications - Feedback Report on the 1998/99 Project Proposals

Contents

1. Introduction

2. Method

3. Ranking the projects

   1. Characteristics of the funded projects

   2. The outright rejects

   3. Those in the middle

   4. Generic Shortcomings

Following the adoption of the White Paper on Science and Technology with its identification of the need for government to support the National System of Innovation, a number of important steps were taken. One of these was to set aside funds to be accessed through competitive bidding. These funds constitute the Innovation fund (IF). The IF is a program of support that addresses problems "serious enough to impede socio-economic development or affect our ability to compete in products and services". The funds are intended for large-scale projects and should involve a significant component of R & D. They should generate new knowledge leading to novel products, processes or services.

The first limited round of IF funding took place in 1997/98 and targeted the area of crime prevention. In the 1998/99 financial year a wider process was run with R30 m available to the bidders. The funds were now targeted at:

• Crime Prevention
• Promoting the Information Society, and
• Value-addition

A total of 177 proposals were subjected to review by independent technical evaluators. The final selection was made by three groups of stakeholders drawn mainly from the major government departments, both national and provincial.

There are many lessons to be learnt from the process in order that the next funding round may be performed with greater efficiency and effectiveness. This brief study is intended to deliver recommendations in terms of generic shortcomings evident in the proposals and to suggest the manner in which they might be rectified. 

Top of page

There were three main components to this work. Consideration was given to:

• The role played by the Department in managing the process;
• The technical quality of the proposals;
• The work done by the technical evaluators (TEs).

The quality of the proposals was assessed in terms of the guidelines and criteria for the IF that was laid down by the Department. The efforts of the evaluators were considered against the task expected of them: to evaluate proposals against a range of technical and social criteria, and to do so with accuracy, consistency and objectivity.

The evaluation reports of the technical evaluators were read and key points extracted to a database. The scores for each of the six designated categories were also recorded.

A total of 177 projects were considered for funding under the review. A total of 19 awards in the amount of R38,5m, were made as follows:

• Crime Prevention (R12m) (project numbers starting with a "1")
• Promoting the Information Society (R11m)  project numbers starting with a "2");
• Value-addition (R15,5m) (project numbers starting with a "3"). 

Top of page

This meta-evaluation presented the interesting challenge of sifting through a large volume of technical evaluation reports in a limited space of time, to capture the essence of the decision-making process. Faced with such a large task it was therefore decided to adopt a triage approach. First to consider the findings of the Technical Evaluators in respect of the successful projects; second to consider the characteristics of those rejected outright, and finally to examine those projects in the middle that might have gone either way.

1. Characteristics of the funded projects

It is clear that the single most important criterion for acceptance by the selection panel relates to the perceived novelty and innovation potential of a project. The comments made regarding the type and degree of innovation range from statements such as "it is just brilliant" and "this is breakthrough innovation" to "novel product", "promotes innovation through research", and "a high level of innovation is shown in taking existing technology and applying it to the proposed use". The evaluators were less concerned with the extent of innovation, but rather more concerned with application to new areas provided this addressed the social objectives set by the IF in respect of crime prevention, promoting an information society, and contributing to value addition. The message implicit in this selection is: capture the imagination of the IF assessors and understand the agenda of government. After all, these are government funds that are being deployed.

In the views of the TEs the criterion next most critical was the capability of those proposing to do the work. Unsurprisingly those consortia that were successful were regarded as "strong", "exhibiting all the characteristics of the linkages that the IF seeks to promote", "representing excellent teamwork between industry and academia", and "drawing on excellent expertise".

The next most important criterion was to do with the market. The successful projects and the associated comments of the evaluators indicate an understanding of the dynamics of change and time to market. Benefit to the country in terms of new products and to the previously disadvantaged in terms of a raised quality of life takes time to filter through. Provided the evaluators were convinced that a market or need for the innovation existed, there was a generosity of spirit shown toward the projects. After all, no proposal is likely to be perfect. Regarding market possibilities, successful projects were characterised as "the project creates a new niche industry", "they have done their homework on the market" and "there is high commercial applicability".

Whilst not entirely uncritical of the budgets proposed by those selected for funding, it should be obvious to applicants that the scrutiny their proposals were subjected to, was executed by peers who knew how to read a business plan. Public funds are scarce, and where there was suspicion of "double dipping" or inflation of costs, it was sternly resisted.

As regards which projects were funded (Table 1), it is the case that of the nineteen awards, all but two went to consortia of three or more parties, with ten of the latter including a number of higher education institutions. It is evident that few large private sector corporations submitted bids, and that of this small number only one was successful. The distribution across universities and technikons was more or less as expected, although their participation (36) was even more widespread than the Science Councils (27 occurrences, with CSIR = 18). Eighteen private sector entities, including various Denel components also participated.

Table 1: Recipients of IF funding – consortium participants.

 

It is useful to consider some basic numerical data that allows for comparison of the recipients to the total group of proposals.

 

Table 2 presents the average ratings for the six categories of the IF for the successful recipients as opposed to the total group. It is quite clear that they stand apart. However, this observation must be treated with care since the two group scores are created by different parties. They may have used the same "instrument" but are not a common group. Moreover, it is the case that the Technical Evaluators themselves differ in their ranking of the projects that received awards.

Table 2: Profile of Recipients across six categories compared with total Group. 

 

The occurrence of maximum scores in the six categories was also examined as a check to determine whether the recipient projects were really significantly different to the group as a whole, and also to check on the extent to which the recipient projects were seen to be excellent. This was done for both groups (Table 3) and expressed as a percentage.

Table 3: Frequencies of maximum scores between Recipients and Group. 

 

The chi test on this data is significant at better than the 0,0001 level.

It is therefore the case that there are clear winners. So what makes for a winner? Capture the imagination! Show where the innovation is; work from strength by having a real and broad-based consortium of players. Understand the agenda of government in order to bring forward proposals that make sense. Ask the questions: what is likely to work in the social market place, and what are the major thrusts in the areas where the IF is focusing? Who will the major beneficiary be and what contribution is likely in respect of redress? 

Top of page

 

2. The outright rejects

The totally rejected projects may be best described through the words of their reviewers. The group at the lower range of the scores is represented in the following sample statements:

• scanty on detail
• not ready for evaluation
• no funding detail
• poor market research
• request for start-up funds or venture capital
• ill-conceived and totally unrealistic
• a weak understanding of technology in its social context
• consortium is just individuals without relevant expertise
• narrowly focused without links to potential users or necessary funds
• no more than a technical exercise with poor appreciation of risk.
• highly implausible market returns
• benefit to the previously disadvantaged unlikely
• main beneficiary is the proposer.

Other telling comments are: "an astonishingly lightweight consortium", "a random collection of projects;" and "no evident innovation content or real project plan". The TEs appear to have been sensitive to opportunistic applications that saw the IF as a source of easy money that might be accessed even if the proposers had no direct experience in the field being proposed and had cobbled together a proposal without even a cursory literature review.

Other major weaknesses were in the area of market research, potential for commercialisation, and links with possible users. Worse were cases where the technologies already existed and the proposers had apparently not even checked the Internet to cover that possibility.

What we may conclude from this analysis is that the evaluation framework did succeed in sifting out the quality from the chaff. The framework may not be perfect, but it has merit in its simplicity. 

Top of page

 

3. Those in the middle

The group of applications "in the middle" are perhaps those where the applicants would be most concerned with wishing to know why they were unsuccessful. For this group one may attempt to construct advice based upon the experiences of the extreme groups: what to seek and what to avoid. It is thus useful to spend some more time on the work of the TEs and what the IF intends to achieve.

Some aspects of the way that the TEs worked have already been considered. In order to elicit some patterns, correlations were sought between the different scoring categories and the total scores. These correlation coefficients suggest that high overall scores depend in sequence upon:

• The project
• National benefit
• The Consortium
• Management
• Benefit to the disadvantaged and market need.

This suggests that the project must provide a self-evident component of innovation. If it is an industrial innovation, then where does it fit into the standard OECD schema? If it is a social innovation, where is the real novelty? OECD is rather precise on these matter, classifying surveys and routine software development development as S & T services, not R & D. The lesson here is that most attention is placed on the project itself. This is the "what" question.

The project itself must be regarded as feasible and necessary in relation to national benefit. For the proposers to get this right it would be necessary for them to have a good sense of the priorities that government has set for itself. Knowledge of policy positions or the stances adopted by government-industry bodies will help here. This is the "why" question.

It would appear that the TEs then considered the "who" question. Who will do the work, what is their track record, and do they possess the necessary skills to make it happen? Only after that is the "how" question looked at. In a sense this closes off the technical aspects of the project, representing the operational aspects of the work. A major reason for disqualification was the perceived weakness of research skills, a weakness that came through in the written proposals themselves. Many of the TEs are academics, and required convincing that proper literature reviews had been done. After all, why re-invent the wheel at taxpayers’ expense?

After that the TEs consider the "and then" questions of benefit, market return and so on, which are all questions of the longer term. This was especially difficult for those projects emanating from the social sciences, where "market" is a fuzzy concept, and may be replaced with "dissemination". The TEs were not convinced that collecting data represented a good use of the IF. 

Top of page

 

4. Generic Shortcomings

Implicitly, what the IF is doing is to conduct a "request for proposals" (RFP) process against a very widely defined canvas. The group of potential bidders is unused to this type of arrangement and does not necessarily understand what the IF wants. It is also the case that the first set of proposals displayed generic shortcomings across a wide range:

• Misunderstanding the IF purpose
• Choice of project
• Recycling existing project proposals
• Weak skills in proposal writing
• Inadequate prior research
• Weak or non-existent consortia
• Attempting to use the IF to allow existing solutions to find a problem
• Weak project planning skills
• Weak research design
• Failure to include milestones
• Inadequate consideration of impact assessment
• No consideration of redress criteria
• Poor budgeting skills
• Inflated budgeting
• Weak understanding of the route to market
• Confusion regarding the meaning of innovation.

Top of page