ISSN 0798 1015

Vol. 39 (Number 44) Year 2018. Page 22

The development of a multi-criteria approach to assess innovative projects efficiency

Desarrollo de enfoque multicriterio a la evaluación de eficacia de los proyectos innovadores

Tatyana V. ALEXANDROVA 1; Svetlana L. ZHUKOVSKAYA 2; Nikolai Yu. VOEVODKIN 3

Received: 10/05/2018 • Approved: 12/06/2018

Contents

1. Introduction

2. Methodology

3. Results

4. Conclusions

Acknowledgments

Bibliographic references

1. Introduction

The creation and commercialization of innovations is a key development factor in the modern economy and a necessary condition in raising organizational competitiveness.  Due to diffusion and synergy effects in innovations, the implementation results at separate enterprises influence innovative projects in industries, regions, national and world economies.

According to L. Lengran (Lengrand, 2006, p. 82), it is essential to assess the potential of an innovative project really, as it has a significant influence on the processes aimed at regional and state innovation policy development, choice of the national innovative system’s development priorities. Therefore, it increases the responsibility of innovative activity participants for the results of choosing projects for further financing and implementation. This influences the choice of investors, model of financing innovations, perspectives of using innovative products and technologies.

It should be noted that in practice the majority of enterprises and investors rather formally assess the efficiency of an innovative project. Usually the analysis is conducted during the decision making process to assess the purposefulness of innovative project implementation, which includes the study of the enterprise’s – the project recipient’s – financial state; the calculation of standard commercial and economic efficiency indicators; and the evaluation if the company satisfies the bank requirments on the client’s credidability (Agafonova, Chelak, 2009, pp. 30 –33; Barilenko, Berdnikov & Gavel, 2015, pp. 110 –162; Kozlovskaya, Dimidenko & Yakovleve, 2012, pp. 245 –310).

Nowadays there are many publications about assessing innovative projects efficiency. It is considered that a classical approach to assess the innovative activity uses mostly financial indicators, showing the return of investments in creating and commercializing innovations. This approach calculates generalizing indicators, characterizing the efficiency of an innovative project in general; as well as special ones, reflecting the efficiency rate of using certain kinds of resources in the innovative activity. These form the basis of «Methodological recommendations to assess innovative projects efficiency and choose them for implementation», which regulates the assessment process of investments and innovations’ efficiency in the Russian Federation (Methodological recommendations, 2014, pp. 120 –186).

More modern models to assess innovative activity, which appeared in the late XXth century, attracted attention in both Russian and foreign economic literature. These models were based on assessing a broader range of innovative project’s parameters; they linked different aspect of innovative activity, orientated the innovation assessment process on achieving strategic goals in running an enterprise (Kaplan & Norton, 1996, pp.90 –110; Gunin, 2000, pp. 114 – 156; Endovitskiy, 2004, pp. 198 – 243; Zaslavskiy, 2014, pp. 210 –222).

Studies are also conducted to systemize existing methods of assessing innovative projects’ efficiency. A special attention should be paid to the classifications which consider methodological approaches to innovation efficiency assessment from evolutional point of view (Yashin, Yashina & Koshelev, 2010, pp. 42 – 49; Gadzhiev and Yakovleva, 2010, pp. 122 – 126; Tsvetkov, 2012, pp. 1689 – 1693).

Table 1 represents the classification of current methodological approaches to assess innovative projects efficiency, created by the authors. As the table shows, the multi-criteria approach is a modern and perspective technology to evaluate the project activity efficiency. It forms the basis for transition to integrated assessment models, and different variations of the balanced indicators system are its vivid examples. A multi-parameter approach lays a foundation for the development of «Oslo Manual» (Oslo manual, 2005, pp. 18 – 20), which contains recommendations for gathering and measuring data in the sphere of innovations for the member-countries of the Organisation for Economic Co-operation and Development (OECD).  However, this document does not have the list of specific indicators, which can be used to assess the innovative projects’ efficiency.

Table 1
Methodological approaches to assess the innovative projects’ efficiency

The global reorientation of the world’s economy on innovative development requires further research in the area. One of the topical issues is how to provide a complex assessment of the innovative project’s efficiency, taking into account its specific features. In particular, the following is demanded:

• to specify innovative projects;
• to use the theoretical potential of multi-criteria approach in assessing projects’ efficiency to a greater extend;
•  to include more indicators into groups of innovations’ efficiency criteria;
•  to specify the acceptable values interval in each criterion to identify the overall quantitative estimate of a project; 
•  to work out recommendations to choose projects for financing based on final estimates, obtained with the given list of criteria.

This constitutes the topicality of the issue, as well as the research goal and tasks.

The purpose of the study consists in developing theoretical and methodological issues, connected to using the multi-criteria approach in assessing the innovative products’ efficiency.

To achieve this goal the following tasks had to be solved:

• to formulate innovative projects specific features, which have to be taken into account while making the list of efficiency assessment criteria;
• to choose the combination of methods to estimate the project’s efficiency, which increases the justification of making a decision using the multi-criteria approach in the innovative management of organizations;
• to identify the list of indicators to assess the efficiency of innovative projects and to specify the estimation interval for each indicator;
• to structure the innovative project efficiency assessment process;
• to work out typical variants of managerial decision making as the result of multi-criteria assessment of the innovative project’s efficiency.

The theoretical basis of research consists of the works by Russian and foreign researches in the innovative and investment management, proprietary technologies to assess innovations’ efficiency, and legal documents regulating the process of innovative projects efficiency assessment.

The empirical data was taken from the Russian Federal Statistics Service, the results of statistical research by the State National University «The Higher School of Economics», and the information about the activity of the creative business incubator «Mozgovo» of the Perm State National Research University.

The table 2 demonstrates the sample of statistical indicators of innovative activity in Russian organizations, made by the authors based on the Federal Statistics site.

Table 2
The indicators of innovative activity in Russian organizations.

Despite the extensive increase in costs for innovations, the main innovative activity indicators demonstrate a downward trend. For the number of years there has been a decrease in the share of organizations implementing innovations in the total number of organizations (the innovative activity of organizations), including the companies implementing technological innovations. Moreover, the Russian rate of innovative activity lags behind the western economies, for example, this rate exceeded 50% in some developed European countries in 2016 (Indicators, 2018, p. 314).

Considering the cost intensity for innovations, this indicator correlates with the European average and even exceeds it (Indicators, 2018, p. 324), despite the decrease in the share of costs for technological innovations in the total amount of shipped goods (from 2,9% in 2013 to 2,5% in 2016).

One of the indicators showing the productivity of efforts to develop innovations is the share of innovative goods, works, services in the total amount of shipped goods, work done, and services provided. This indicator has a downward trend, it decreased from 9,2% in 2013 to 8,5% in 2016. In addition, it is much lower than the same indicator in some developed European countries (Indicators, 2018, p.326).

Global Innovation Index, which is calculated by the World Intellectual Property Association (WIPO) UNO, allows to generally assess the efficiency of the national innovation system. This index is the ratio of costs and effects of innovations. Russia shows a stable increase in the sub index of innovation resources (the 43th place), but the country has a weaker position on the efficiency of innovative activity (the 75th place). In the final rating Russia occupied the 45th place out of 127 (The Global Innovation Index 2017). 

The Russian dynamics in the Global Innovation Index rating is shown in the Table 3.

Table 3
The Russian dynamics in the Global Innovation Index rating.

Innovatively developing economies differ by the biggest share spent on their own research and development and the smallest proportion spent on purchasing ready-made innovative decisions. Despite the positive dynamics (the share for own research and development increased from 20,4% in 2012 to 23,6% in 2016) the largest proportion of it is spent on machinery and equipment (53,2% in 2016) (Indicators, 2018, p.49). 

Having studied innovative activity in national organizations, the economists of the National State University of the Higher School of Economics singled out thirteen factors, preventing the development of Russian own innovations (Indicators, 2018, p.57). Economic factors have the highest rating, for example, insufficient funds (rating 2.2), high cost of innovations (2.2). The second group is internal factors, which include a low innovative potential of an organization (1.7), insufficient qualified personnel (1.6) and others. The factor of an uncertain economic benefit from using intellectual property has a high rating in the third group of factors (1.7). 

The statistical analysis resulted in three research hypothesis: 

• Development and implementation of the own innovative projects is a more complicated activity than purchasing ready-made innovative decisions, worked out by third parties and implementing traditional business projects.
• The efficiency of innovative activity in organizations can be increased using a more justified and multi-aspect methodology to economically support and choose innovative projects for financing and implementing.
• A practical application of the improved methodology to assess innovative projects’ efficiency and to choose them for financing and implementation contributes to improving innovation activity in organizations in general.

The methodology of research is based on critical thinking methods, allowing to emphasize problem areas; scientific methods, using multi-criteria approach to assess innovative projects’ efficiency; «Oslo Manual» methods, recommending to increase the number of innovative measurements in innovative organizations.

A combined variant of criteria listing method was used to develop the methodology of the innovative project efficiency assessment and making a decision about its practicality. It includes the elements of the innovative project’s economic efficiency assessment, points and expert methods, and the ones of risk factors consideration.

There is no unified approach to innovative efficiency assessment in both modern theoretical and practical innovative management; the standard methodology to estimate the practicality of choosing innovative projects for financing is not developed. There is an objective reason for such situation, as the process of innovative projects’ quantitative assessment is more complicated compared to usual investment ones, for which there is a developed methodology to assess efficiency based on profitability and return on investments indicators.

The authors believe that the specific features of an innovative project, identifying the necessity of a special methodological approach to assess its efficiency, include:

• Only innovations are considered as investment objects. The novelty rate and commercial perspectives must be taken into account in the assessment process.
• The state can act as a co-investor through the system of grants, non-budget funds, federal and regional programs to develop science and innovations, and others. State demands to innovation profitability must be considered while assessing projects’ efficiency.
• Innovation projects are characterized by a high risk, uncertainty during the whole cycle, therefore a special attention must be paid to identification, and risk calculation procedures at all stages of the innovative project’s development, production and implementation.
• Choosing alternative variants of managerial decisions remains an actual issue for innovative projects throughout their lifecycles, even after they were chosen for financing. Qualitative parameters often reconsidered and specified. Several calculations have to be made, methods of modeling and predicting cash flows are used while assessing innovations efficiency.
• The innovative product’s lifecycle has the most influence on the project’s lifecycle. The shorter the innovation’s lifecycle is, the faster the assessment procedure must be and the decision about the practicality of its implementation must be made.
• An innovative project is characterized by a large amount of information, which is not included into economic calculations. However, this information has a significant influence on a decision making process about the practicality of its implementation. That is why the quantitative assessment must be accompanied by the qualitative analysis of separate project parameters, identifying subjective preferences of leading participants (Levis, 2011, р. 21).
• An innovative project has a more complex success criteria, than a usual investment one. In the assessment process an important place is occupied by such criteria as the product’s novelty, patent’s clearance, license protection, the experience of the recipient company on the new products’ market, the influence on priority directions of scientific and technical progress and others. 
• More effective measures must be taken to stimulate innovative projects’ participants materially, because they will invest in the situations of increased risks. Therefore, a higher rate of return must be put behind while calculating economic efficiency indicators of an innovative project.
• Economic profitability of many innovations has a postponed strategic character; it is not predictable enough and often can be estimated only within financial improvement and a general competitiveness of the company.

This suggests that it would be incorrect to project standard methods of investment efficiency assessment and the analysis of the enterprise’s financial state fully onto the assessment system of innovative project’s efficiency parameters. A special methodology must be used to estimate this efficiency, which takes into account specific features of creating and commercializing innovations, the structure of innovative process in an enterprise.

It is recommended to use a combined variant of the criteria listing method in solving the problem of efficiency assessment; and five criteria groups can be singled out:

• Social criteria;
• Economic criteria;
• Market criteria;
• Investment efficiency criteria;
• Scientific and technical criteria.

Specific indicators are identified in each criteria group; they are estimated by the experts who are highly qualified specialists in innovative activity. The experts give a project scores from 0 to 3 depending on how successful it is by one or another indicator. The higher the score, the more successful is the project by this indicator. According to the suggested methodology, the assessment procedure consists of the following stages: 

Stage 1. Gathering and systematization the information about the innovative project.

Stage 2. Working out the set of indicators to assess innovative projects.

The recommended list of the innovative project risk indicators includes:

1. For the social criteria group

• creating new working places;
• solving the problems of territorial development;
• providing the population with food;
• providing the population with transport and communication;
• influencing on the level of the population’s education and culture.

2. For the economic criteria group:

• increasing the efficiency of using resources;
• increasing the products’ profitability;
• increasing the products’ quality;
• a contribution to the structural reorganization of the territorial economy.

3. Market criteria:

• products’ competitiveness;
• a position in competition;
• demand on the product;
• market risk;
• the risk connected with attracting and financing investments.

 4. Innovation efficiency criteria:

• R&D costs;
• the structure of the capital, formed in the project;
• price of capital;
• investment profitability index;
• rate of return term;
• investment profitability;
• the time when the project starts to make profit.

 5. Scientific and technical criteria:

• project development terms;
• the contribution to the development of other trends in scientific and technical development;
• inclusion into previous programs and projects of the company, industry or territory;
• the availability of highly qualified researches;
• the project’s novelty rate.

Stage 3. Defining the innovative project’s indicators variations with assigning them permissible ratings (refer to table 4). 

Stage 4. Defining the integrated rating of the innovative project based on summing separate indicators ratings.

Stage 5. Identifying the success class of the innovative project for corresponding intervals of the project’s rating; making a decision about the practicality to finance the innovative project.

The summary decision variants by the project’s estimation results:

1. 50 point and more – the project deserves unconditional support. The federal financial support is recommended.

2. 26 – 50 points – the project deserves support. The regional financial support is recommended.

3. 13 – 15 points – the project can be supported. The municipal financial support is recommended.

4.   6 –12 points – the project does not deserve state support, but can be implemented with the help of commercial financial sources.

5.  2 – 5 points – the project does not deserve support and is not worth implementing.

Table 4
Criteria and indicators of efficiency assessment for the innovative project

The table 4 shows the criteria and indicators of the innovative project’s efficiency assessment, developed by the authors as the result of summing up the expert practice of the creative business incubator «Mozgovo» and innovative enterprises in Perm Krai.

The main research results can be used in factories, scientific and technical, and expert organizations by developing innovative projects and justifying their practicality for financing. In 2017 the methodology developed by the authors was tested in the project expertise in business incubator «Mozgovo», which is a part of Perm State National Research University.

Thanks to a more justified project choice in 2017, compared to 2016, twice as many business incubator «Mozgovo» projects became winners of the regional contest for innovative projects «UMNIK». The number of projects, which became winners of the regional innovative projects’ contest «Start», increased by 33%. This contest is conducted annually in Perm Krai for small and medium business enterprises. For the first time venture investments were attracted to finance knowledge-intensive innovations. Also the projects took part in grant competitions of the innovative fund «Skolkovo» and the Fund of the internet initiative’s development (table 5).

Table 5
The influence of the multi-criteria methodology on the change of innovative
activity indicators in the activity of the creative business incubator «Mozgovo»

However, specialists and residents of the creative business incubator «Mozgovo» faced some problems while applying this methodology, alongside with its positive effect. They are linked to the fact, that the procedure has become more complicated, many-sided and labor-intensive. The specialists required a higher level of qualification, deeper knowledge about the innovative process, the skills to work not only with quantitative, but also with the qualitative parameters of the innovative project.

The expert board has been created to solve the problem of knowledge deficiency, which is a consulting body. The Perm State National Research University prepared special bachelor, master and post-graduate programs «Innovative entrepreneurship» to disseminate knowledge about innovations, special features of their development, economic justification and commercialization. The study program «From students to residents» was created for the business incubator residents.

The study results allowed to single out specific features of innovative projects, which complicate significantly their efficiency justification process as well as choosing them for financing. This confirms the first hypothesis.

 It justifies the necessity to develop a multi-criteria methodology for assessing the efficiency of innovative projects, which focuses on a broader range of innovations assessment’s parameters, and taking into account different aspects of the innovative activity in an organization. It analyses the influence of the new technology to assess innovative projects on innovative activity indicators of the creative business incubator «Mozgovo». The analysis results demonstrate a significant increase in the innovative activity of the organization under study, that proves the second hypothesis.

As for the third hypothesis, it confirmed partly, it means that the increase in justification of making innovative decisions does not only improves the organization’s activity, but also creates new problems, linked to using new knowledge and skills in practice. The company’s managers should identify these problems quickly and take purposeful measures that the staff develops new skills.

Using the suggested methodology can help to formalize the managerial decision-making process about the practicality of choosing the innovative project for implementation. This method will provide a more argumentative choice of financing variations in favor of the certain innovative project, if it corresponds to the criteria of innovation’s success, the rate of the company’s readiness to future innovative activity, and the innovation’s importance for the regional and national socio-economic developments.

Acknowledgments

The authors are grateful to Nikolai Nikolaevich Kosvintsev – the head of the creative business incubator «Mozgovo» at Perm State National Research University for practical advice and valuable comments.

Agafonova I.P., Chellak S.L. (2009). The technology of assessing risks and rating the innovative project when organizing its bank financing. Financial management, 6, 28 - 39

Anpilov S.M., Bezlepkina N.V., Tjukavkin N.M. (2011). Economy and management in the XXI century/ Under the general editorship of Saraev L.A. (p. 231). Samara.

 Barilenko V.I., Gavel O.Yu., Kerimova Ch.V. (2015). Informational and analytical methods to assess and monitor the innovative projects efficiency. (p.163). Moscow: Ruscience.

Davila T., Epstein M.  Shelton R.  (2012).(p.368).Making Innovation work. FT Press.

Gadzhiev M.M., Yakovleva E.A. (2010). The analysis of the enterprise innovative activity’s economic efficiency. Innovations, 2, 122 – 126. 

Gault, F. (2013) Handbook of innovation indicators and measurement. (p.486) Cheltenham: Edward Elgar Publishing.

Gunin V.N. (2000) Managing innovations: 17-module program for managers «Managing organizational development» (p.272). Moscow: INFRA-M.

Innovative Activity Indicators (2018). (p. 344). Moscow: NRU HSE. Retrieved from: https://www.hse.ru/primarydata/ii (accessed Маrch 11, 2018)

Indicators of Innovation in the Russian Federation:2018: Data Book; National Research University Higher School of Economics (2018). Moscow: HSE. Retrieved from: https://www.hse.ru/en/primarydata/ii (accessed Маrch 12, 2018)

Kaplan, R.S., Norton, P.D. (1996). The balanced scorecard: Translating strategy into action. (p. 336) Boston:  Harvard Business School Press. MA.

Kozlovskaya E.A., Demidenko A.S., Yakovleva E.A. (2012). The economy and innovation management. (p.359)/ Under the general editorship of E.A. Kozlovskaya. Moscow: Economics.

Lengrand L. (2006). Smart innovation: A practical Guide to Evaluating Innovation Programmes. (p. 199) Brussels-Luxemburg: University of Manchester, ANRT.

Lewis D.A. Molnar, Elsie Harper-Anderson, A. Lawrence (2011). Incubating Success: Incubation Best Practices That Lead to Successful New Ventures.(p. 223) University of Michigan.

Materials of the Federal State Statistics Service of the Russian Federation Retrieved from: http://www.gks.ru/wps/wcm/connect/rosstat_main/rosstat/ru/statistics/science_and_innovations/science/ (accessed April 12, 2018)

Methodological guidelines on assessing innovative products’ efficiency and choosing them for investment. Official edition. (2004). (p.278). Moscow: Economics.

Organizing analysis and control of innovative activity in a company. (p.272). Moscow: Finance and Statistics.

Oslo manual. Guidelines for collecting and interpreting innovation data. 3th ed. Paris, OECD/EC, 2005. (p.210)  Retrieved from: https://mgimo.ru/upload/docs_6/ruk.oslo.pdf 9 (accessed Маrch 12, 2018)

The Global Innovation Index 2015. Effective Innovation Policies for Development.  Geneva; New Delhi: Cornell University, INSEAD, and the World Intellectual Property Organization. Retrieved from: http://www.globalinnovationindex.org/ (accessed March 23, 2017)

The Global Innovation Index 2016. Winning with global innovation.  Geneva; New Delhi: Cornell University, INSEAD, and the World Intellectual Property Organization. Retrieved from: http://www.globalinnovationindex.org/ (accessed March 23, 2017)

The Global Innovation Index 2017. Innovation Feeding the World.  Geneva; New Delhi: Cornell University, INSEAD, and the World Intellectual Property Organization. Retrieved from: http://www.globalinnovationindex.org/ (accessed March 10, 2018)

Tsvetkov V.Ya. (2012) Innovations’ analysis according to OECD standards. European researcher, series A, v. (31), 10, 1689-1693.

Yashin S.N., Tukkel I.L., Koshelev E.V., Makarov S.A. (2018) The assessment of innovative activity efficiency/ Under the general editorship of S.N. Yashin, (p. 312). Nizhniy Novgorod: The Nizhniy Novgorod University Publishing House.

 Yashin S.N., Yashina N.I., Kishelev E.V. (2010). Financing companies’ innovations and investments. (p. 245). Nizhniy Novgorod: Publishing house VGIPU.

Žižlavský O. (2014). Net present value approach: Method for economic assessment of innovation projects. Procedia Social and Behavioral Sciences, 156, 506–512. 

Žižlavský O.  (2014.) The balanced scorecard: Innovative performancemeasurement and management control system. Journal of Technology Management and Innovation,  9 (3), 210–222.  

1. Associate Professor, Candidate of Economic Sciences, Department of Management, Perm State National Research University, Russian Federation, e-mail: atvpsu@yandex.ru

2. Associate Professor, Candidate of Economic Sciences, Department of Accounting, Audit and Economic Analysis, Perm State National Research University, Russian Federation, e-mail: svlvzh@yandex.ru

3. Associate Professor, Candidate of Philological Sciences, Department of Management, Perm State National Research University, Russian Federation, e-mail: voevodkin_n@mail.ru