Industrial Technologies Office
ito.ic.gc.ca
Common menu bar links
SADI RMAF/RBAF
Program Profile
2.1 Context
2.2 Program Description
2.2.1 Project Life Cycles
2.2.2 Repayments
2.3 Program Objectives
2.3.1 Link to Program Activity Architecture
2.4 Stakeholders and Beneficiaries
2.5 Program Delivery
2.6 Roles, Responsibilities, Relationships
2.7 Resources
2.8 Overall Risks
Logic Model
3.1 Logic Model Summary
3.1.1 Short Term Outcomes
3.1.2 Intermediate Outcomes
3.1.3 Ultimate Outcome
3.2 Internal and External Factors
3.3 Logic Model
3.4 Accountabilities
Risk Assessment and Management Summary
4.1 Methodology
4.2 Risk Management Team
4.3 Risk Identification, Assessment and Mitigation Strategies
4.4 Risk Matrix
4.5 Key Risks
Monitoring and Evaluation Plan
5.1 Performance Measurement Plan
5.1.1 Audit Plans
5.2 Evaluation Plan
5.2.1 Evaluation Issues
5.3 Reporting Commitments
Annex A: Project Results Monitoring
Introduction
This is an integrated Results-based Management and Accountability Framework (RMAF) and Risk Based Audit Framework (RBAF) for the Strategic Aerospace & Defence Initiative (SADI) program. The document addresses all requirements identified in the Treasury Board Secretariat (TBS) guidelines for RMAFs as well as the guide to the development of RBAFs. It is developed according to the recommended structure for integrated RMAF-RBAF documents, as per Annex D of the January 2005 guide Preparing and Using Results based Management and Accountability Frameworks.
This integrated RMAF / RBAF deals essentially with the management, accountability and risk factors related to the SADI program. The program is to be administered by Industry Canada (IC) through the Industrial Technologies Office (ITO), which is a special operating agency of IC. The RMAF / RBAF formalizes the commitment of the ITO to ensure that the SADI program has a strong accountability framework to measure progress and demonstrate results. It also clarifies the anticipated outcomes; mechanisms to measure results; and, anticipated reporting methods.
Program Profile
2.1 Context
Canada's aerospace, defence, space and security (A&D) industries are recognized around the world for producing leading edge products and services. Domestically, the industries are important economic drivers in all regions of the country and, collectively, are a significant player in the development of Canada's knowledge-based economy.
A&D industries are the third largest investor in R&D, accounting for 11% of all Canadian industrial R&D. Public support ensures A&D firms undertake R&D in Canada (especially strategic projects) and that the country benefits from spin-offs from these R&D investments. Supporting R&D in the industry also enables Canada to attract leading global A&D firms (increasing/retaining foreign direct investment) and world-leading scientific and engineering talent in high-technology A&D industries to Canada.
To address the importance of investing in R&D, the SADI program will support corporations incorporated under Canadian law and that are prepared to conduct strategic R&D activities in A&D industries. For the first 5-year time horizon of the program, SADI will approve nearly $900 million in funding for multi-year projects.
Given that the Canadian A&D sector is large – it had sales of $21.8 billion, employment of 75,000 highly-skilled Canadians and R&D investment of nearly $1.2 billion in 2005 – the overall extent to which the SADI program can impact the A&D sector as a whole will be limited.
This document outlines the management and accountability framework that will be used to measure SADI progress and to demonstrate results. It describes the anticipated outcomes from the program, the mechanisms to measure results and the anticipated reporting methods.
2.2 Program Description
SADI is a new A&D initiative that is designed to support strategic R&D. Strategic R&D activities are those that:
- Support the development of next generation A&D related products and services;
- Build on existing Canadian strengths in A&D technology development;
- Enable Canadian companies to participate in major platforms and supply chains; or
- Assist A&D industries in achieving Canada's international obligations (e.g. development programs supported by Canada).
SADI will provide repayable contributions to recipient companies for eligible costs incurred and paid in respect of industrial research and pre-competitive development in A&D carried out by that recipient company in Canada. The level of assistance from all government sources (federal, provincial, territorial, municipal) to an eligible recipient shall not normally exceed 75% of eligible costs. The Minister may approve, on a case-by-case basis, an exemption to the stacking limit.
2.2.1 Project Life Cycle
Once a contribution agreement is signed with a company, a project goes through two phases (R&D phase and benefits phase) that can last up to 20 years. The R&D phase is expected to last 5 years on average, during which the company works on its technology research and development. A target of 15 years, on average, has been set for the benefits phase, which is the period when repayments are due to the Crown.
2.2.2 Repayments
Recipient companies are expected to make repayments to the Minister as a condition for receiving funding support under the SADI program. SADI contribution agreements will stipulate the obligations on the part of recipient companies with respect to repayments. Repayment plans will follow a standardized approach based on a company's gross business revenues and will not be based on revenues derived solely from project-related products, processes and services. Repayments will begin approximately one year after the completion of the R&D phase of the project and will have an average repayment period of 15 years.
2.3 Program Objectives
SADI will foster the growth of a competitive, knowledge-based Canadian economy. SADI supports the IC's Strategic Objective: an innovative economy.
Specifically, the objectives of SADI are:
- Encourage strategic R&D that will result in innovation and excellence in new products and services.
SADI's intent is to act as a catalyst for new A&D investments in defined strategic R&D, by providing repayable contributions that will allow companies to leverage other funding. By ensuring that contributions are directed at areas strategic to A&D sector growth the expectation is that the existing technology knowledge base will be enhanced.
All projects must demonstrate the ability to achieve this objective.
- Enhance the competitiveness of Canadian aerospace, defence, space and security (A&D) companies.
SADI contributions will assist a company to carry out industrial research and pre-competitive development; such R&D may result in a company becoming more competitive.
All projects must demonstrate the ability to achieve this objective.
- Foster collaboration between research institutes, universities and colleges, non-profit organizations, and the private sector.
SADI will increase collaboration both in terms of building new relationships and of encouraging more collaborative relationships between organizations. Degrees of collaboration may range from informal cooperative relationships to relationships that involve coordination between the parties (i.e., sharing of information, defined roles and some shared decision making), or to more formal relationships (i.e., involving shared ideas and resources, consensual decision making and frequent communications).
The development of collaborative relationships is intended to benefit university and college students and post secondary institutes as well as recipient companies. Students will benefit from direct involvement in private sector R&D projects leading to an enhanced educational experience and improved employment opportunities. Post secondary educational institutes will benefit from an enriched curriculum that provides opportunities for research in real world applications and which can attract the best and most qualified students. Recipient companies benefit from being able to leverage leading edge technology development ideas being generated in universities and colleges, from being able to have bright graduate students participate in R&D projects and from being able to be more attractive employers for future graduates.
It is recognized that not all projects will have a collaborative element to them as a large number of SADI projects are expected to originate from SMEs that will have limited resources and capabilities to develop, manage and benefit from these types of relationships.
2.3.1 Link to Program Activity Architecture
SADI will advance and support government initiatives by contributing strategically to R&D in the A&D industries in order to encourage and further leverage private sector investment, and so maintain and grow the technology base and technological capabilities of Canadian A&D industries. A key benefit of SADI will be that highly qualified personnel involved in R&D and in the manufacturing of high value-added products will be employed in Canada and will contribute to knowledge diffusion within Canada.
This initiative is linked to the department's objectives and Program Activity Architecture (PAA) as follows:
- Strategic Outcomes – An innovative economy
- Program Activity – Industrial Technologies Office – Commercialization encouraged through strategic investment in innovative research and development
- Sub-Program Activity – Strategic Aerospace & Defence Initiative – Strategic investments in pre-competitive A&D development projects for the economic, social and environmental benefit of all Canadians.
2.4 Stakeholders and Beneficiaries
Canadians, as represented by the government will be the key stakeholder. Other government departments who have a role in supporting A&D industries, including the Department of National Defence, Transport Canada and industry portfolio departments and agencies, such as the National Research Council and the Canadian Space Agency, will also be actively involved in SADI as stakeholders.
The key beneficiaries of the SADI program are Canadian A&D companies (including small and medium sized enterprises). Highly skilled workers in these industries, including future workers enrolled in colleges and universities, will also benefit from the opportunities created by SADI projects. Suppliers to A&D industries and research institutions, university and colleges, non-profit organizations and the private sector will also be beneficiaries as they will benefit from the collaborative R&D that will be supported through SADI.
2.5 Program Delivery
SADI will be administered by IC through the Industrial Technologies Office (ITO), a Special Operating Agency (SOA) of IC.
2.6 Roles, Responsibilities and Relationships
ITO has overall responsibility and accountability for delivery of SADI. To support ITO has relationships with several organizations, within the Department and external to Industry Canada, that play important roles. These include;
Internal relationships with:
- Program and Services Directorate (PSD) – Review of project recommendations
- Program and Policy Management (PPM) – Monitoring of projects in the benefits phase
- Legal Services – Legal advice on project and program requirements
- Audit and Evaluation Branch (AEB) – Internal program audits and program evaluations
- Loan and Insurance Recovery Division (LIRD) – Default recovery management
- Aerospace & Defence and Marine Branch (ADMB) – Project assessment and due diligence support
External working relationships with:
- Department of National Defence (DND) – Project assessment and due diligence support (where needed)
- National Research Council (NRC) – Project assessment and due diligence support (where needed)
- Communications Research Council (CRC) – Project assessment and due diligence support
- Canadian Space Agency (CSA) – Project assessment and due diligence support
2.7 Resources
SADI annual contribution funding will increase incrementally over the next 3-years reaching a funding level of $225 million by year three. This funding level, which excludes operating resources, will be funded through a base budget and access to repayments.
Sufficient resources will be allocated in ITO's annual operating budget for the conduct of due diligence, for monitoring and auditing of projects throughout their life cycles and for tracking and reporting technology and economic benefits achieved by the projects as well as the management and administration of SADI, including the management of repayments. A detailed analysis of these requirements has been conducted and is documented in the ITO Business Case. Resourcing levels will ramp up starting in 2007/08 and will peak in 2011/12.
2.8 Overall Risks
The key risks and mitigating strategies associated with the SADI program are discussed in more detail in section 4 of this document. The following summarizes the key risk areas;
- Funding Capacity – There may be insufficient funding to support program demand
- Large Funding Asks – Some applications may involve projects that require significant financial support
- Due Diligence vs Responsiveness to Applicants – Need to ensure that adequate and sufficient due diligence is undertaken while reducing project review and approval timeline by 50%
- Timely provision of information by applicants – Need to ensure applicant responsiveness to ITO requests to provide information
- Performance Outcomes – Assessing long term outcomes will occur beyond the SADI program time horizon
- Lobbyist Compliance – Recipient company compliance with LRA is important to retaining program integrity
- Revenue & Cost Reporting – Recipient company compliance with costs and revenue reporting requirements is needed
- ITO Organization – Implementation of new organization is critical to effective program delivery
- Information Systems – Infrastructure changes are required to support SADI project management
Logic Model
3.1 Logic Model Summary
The logic model depicts the activities to be pursued, the outputs to be produced and the expected outcomes resulting from the delivery of the program. It provides a basis for developing performance measurement and evaluation strategies.
The SADI logic model can be summarized as follows:
Activities: Activities are what the program does. Typically these activities result from the efforts of staff and the utilization of resources. Activities include:
- Communicating various program information;
- Reviewing and assessing submitted project proposals;
- Conducting due diligence on project proposals;
- Seeking Ministerial decisions on whether or not to fund projects that complete the due diligence phase;
- Negotiating and preparing repayable contribution agreements; and
- Monitoring contribution agreements.
Outputs: Outputs are what the program produces arising from its activities. Outputs can be described as the physical evidence (i.e. documents, resources, services, contribution agreements, etc.) a program produces that flow to beneficiaries and others.
Outcomes: Outcomes are the results that arise from the activities and outputs of the program. Outcomes can be described as short term, intermediate or ultimate.
3.1.1 Short Term Outcomes
Short term outcomes occur as the result of the interaction between staff and applicant companies, beneficiaries and recipient companies. Short term outcomes are those that occur during the R&D phase which, on average, is a five year period. Short term outcomes include:
- Awareness of SADI by Canadian A&D firms;
- Upon execution of contribution agreement, recipient companies undertake strategic R&D projects;
- Increased strategic R&D by recipient companies;
- Leveraging of private sector investments in R&D;
- Retained / Increased R&D capacities of recipient companies; and
- Increased collaboration between recipient companies and research institutes, universities and colleges, non-profit organizations and the private sector.
Funding provided through SADI will mitigate the extent to which development costs, long development lead times and long payback periods impact on the propensity of the company to undertake R&D activities. In addition, SADI funding will leverage other private sector and government investments to ensure that R&D activities are conducted in Canada for the benefits of Canadians.
The R&D projects will in turn create opportunities for recipient companies to engage research institutions, universities and colleges, non-profit organizations and other private sector organizations in collaborative relationships and facilitate the attraction and retention of highly qualified employees.
3.1.2 Intermediate Outcomes
Intermediate outcomes occur as a result of changes to a target group (e.g. Canadian A&D companies) influenced by the impacts of the program. These are expected to occur when projects are in the Benefits phase, a period lasting up to15 years or more, after the completion of the R&D phase. Intermediate outcomes include:
- Canadian public better informed of program benefits;
- Increased development of innovative products, processes and services;
- Increased commercialization of innovative products, processes and services; and
- Increased knowledge diffusion among recipient companies and collaborative partners.
SADI contributions will assist recipient companies to carry out industrial research and pre-competitive development, and that this R&D may result in the creation and commercialization of innovative products, processes and services. Projects involving a collaborative element may increase knowledge diffusion among the collaborators.
Repayments to the Minister will also begin during the period of time when intermediate outcomes are expected to occur (See section 2.2.2 Repayments).
3.1.3 Ultimate Outcomes
Ultimate outcomes, which occur in the larger community, can take up to 15-years or more to materialize; and, can be affected by many other influences. The ultimate outcomes expected from SADI are:
- Increased competitiveness of A&D firms; and,
- To contribute to the achievement of broader technological, economic, environmental and social benefits for Canadians.
Supporting the A&D industry contributes to sustaining strong economic performance that in turn contributes to achieving broader socio-economic outcomes for Canadians.
Broader technological benefits can include the extent to which the R&D initiative demonstrates spill over effects that impact on other fields and the patentability of project results allowing for the broader diffusion of technological knowledge.
Sustainable development benefits include broader economic, environmental and social benefits to which the R&D initiative contributes. For example the maintenance and/or creation of jobs, enhanced productivity and/or economic efficiency, improved safety and security, energy efficiency and reduced green house gas emissions and regional economic benefits.
3.2 Internal and External Factors
While the SADI program has been set up to successfully achieve the expected results outlined in the logic model, a number of internal and external factors could hinder the attainment of such results. Such factors include:
- Economic conditions for the A&D sector may worsen and reduce the level of private sector investment in R&D. This may limit the number of strategic R&D initiatives for the SADI program to support.
- Lack of technological progress should hinder the completion of R&D projects.
- Marginal market adoption of innovative products and services would impede economic performance of A&D firms.
3.3 Logic Model
The logic model for the SADI is presented on the following page. This model outlines the performance framework for the overall program. As such, it is not intended that each individual project will contribute to all of the outcomes identified in the logic model.
Logic Model: Strategic Aerospace & Defence Initiative
Click here to download a larger version of the Logic Model: Strategic Aerospace & Defence Initiative.
3.4 Accountabilities
The accountabilities for the SADI program are as follows:| Organization | Accountable For |
|---|---|
| Minister of Industry | Reporting on the activities of ITO to Parliament Approval of funding for Projects |
| Deputy Minister / Associate Deputy Minister of IC | Review of Projects Recommendation to Minister on funding |
| Assistant Deputy Minister of IC | Review of Projects Recommendation to Deputy Minister/Associate Deputy Minister on funding |
| ITO Executive Director | Management and Operations of ITO Review of Projects Recommendation to Assistant Deputy Minister on funding Reporting on ITO activity through Departmental RPP |
| ITO | Eligibility and assessment screening of applicants Due diligence of projects Formulation of recommendation or approval documents Negotiation and preparation of contribution agreements Project performance monitoring Program performance reporting Recipient company audits |
| Program and Services Board | Review of Projects Recommendations to ITO |
| Industry Canada's Audit & Evaluation Branch | Evaluation of SADI program Internal audit of program |
| Recipient Companies | Complying with provisions of contribution agreements Reporting to ITO on project progress Providing the required information / feedback for the recipient company audits Providing the required information / feedback for the evaluation of the program Providing the required information / feedback for the internal audit of the program Making payments as required under the contribution agreement |
Risk Assessment and Management Summary
4.1 Methodology
The methodology used to identify the program risks comprised of the following:
- The program's objectives were clearly defined and elaborated through the logic model outlined in Sections 2 and 3 of this document.
- The risks associated with the program's environment, design, implementation and results of the program were identified and described through consultations with program management and AEB.
- Each risk was defined and assessed in terms of its likelihood and impact with respect to influencing the achievement of the program's objectives.
- Existing mitigating measures were described and the likely impact of these measures on minimizing the undesirable effect was assessed.
- Residual risk management strategies were identified for key risk events.
4.2 Risk Management Team
ITO management has responsibility for risk assessment and the ongoing monitoring and mitigation of program risks. Supporting the management team is an ITO Risk Management Officer who analyzes project risks at a portfolio level using the portfolio risk management system (PRMS). The system is used to assess and monitor risks throughout the life cycle of each project and involves the annual collection, coordination, analysis, evaluation and entry of data into the PRMS. Risks are assessed and updated annually and mitigation strategies developed as part of the risk management function.
ITO has also formed a Risk Management Committee to provide a governance structure and to oversee the portfolio of SADI projects. The Risk Management Committee is chaired by the Executive Director of ITO and its membership is the ITO Management team.
The AEB risk-based audit planning process also supports ITO management by systematically assessing program risks to determine when and in what depth program audits are required. AEB also conducts evaluations of the program at various stages to assess how the program is progressing towards achieving its objectives and outcomes.
4.3 Risk Identification, Assessment and Mitigation Strategies
The following table outlines the risks identified with the SADI program and summarizes the assessment of the risks based on their likelihood of occurrence and consequent impact. Mitigation strategies associated with each risk are also listed.
ITO classification of risk likelihood and impact is based on the following guidelines.
Likelihood of event occurring during program time horizon
Low – the event is unlikely to occur
Medium – the event is likely to occur
High – the event is expected to occur
Impact of event
Minor – operational impacts on program delivery are minimal and no significant impact on program reputation with clients/stakeholders results
Moderate – some operational disruption impacting program delivery and/or some loss of program reputation and credibility with clients/stakeholders results
Severe – significant operational disruption impacting program delivery and client projects occurs and/or program reputation/credibility is negatively impacted in media and/or from stakeholder/client outcry
| Area | Nature of Risks | Likelihood/ Impact |
Mitigating Strategy |
|---|---|---|---|
| 1. Funding Capacity – Insufficient funding to support program demand. | Allocation of funds to existing TPC obligations and dependency on TPC repayment performance | Medium likelihood Moderate impact |
ITO team with extensive cash flow management experience in place SADI repayment approach improves likelihood of SADI project repayments – will reduce risk starting in years three to five Stringent guidelines and processes for repayment monitoring Ensure that decision makers are aware of the impact on SADI funding of any changes to TPC repayments |
| 2. Large Funding Asks – Some applications may involve projects that require significant financial support | Large commitments of SADI funds will exhaust capacity and ability to fund other beneficial projects | High likelihood Moderate impact |
Officer discussions with applicants will set expectations with respect to funding availability |
| 3. Due Diligence vs Responsiveness to Applicants – Need to ensure that adequate and sufficient due diligence is undertaken while reducing project review and approval timeline by 50% | Avoid compromising due diligence in effort to reduce application processing timeframe Avoid being unresponsive to client's business needs |
Medium likelihood Severe impact |
Officer training to ensure evaluation processes are understood and implemented Various support tools will ensure quality proposals and facilitate assessment and due diligence processes Communicate timelines to clients Stringent and multi-level project reviews will evaluate due diligence quality |
| 4. Timely provision of information by applicants – need to ensure responsiveness to requests for information | ITO's ability to meet targeted application processing timeframes is dependent on availability of timely information | Medium likelihood Moderate impact |
Standards to be established for providing missing information needed for a complete proposal Application materials communicate requirement and timelines to ensure efficient processing for both applicants and ITO |
| 5. Performance Outcomes – Assessing long term outcomes will occur beyond the SADI program time horizon | Ability to demonstrate long term and ultimate outcomes | High likelihood Moderate impact |
Performance measurement plan structured to ensure tracking of progress towards long term outcomes Ongoing communications, including Annual Reports, will highlight expected longer term benefits |
| 6. Lobbyist Compliance – Recipient company compliance with LRA | Recipient companies fail to comply with applicable laws with respect to lobbyist compliance ITO fails to fulfill its Federal Accountability Act obligations |
Low likelihood Moderate impact |
Programs to ensure awareness with applicants and ITO staff Recipient company certifications and contractual terms to confirm compliance ITO procedures to meet internal responsibilities Comprehensive recipient company compliance audit program |
| 7. Revenue & Cost Reporting – Recipient company compliance with costs and revenue reporting requirements | Recipient company claim costs and repayments do not comply with CA terms | Low likelihood Moderate impact |
Programs to ensure claims and repayments processes are understood and that systems to substantiate reporting are in place CA terms and conditions Comprehensive cost and revenue audit program |
| 8. ITO Organization – Implementation of new organization critical to effective program delivery | Delays in operating resource approvals and organizational implementation impacts staffing and staff retention, creates HR issues and impedes development of capability to deliver program | Medium likelihood Moderate impact |
TB Submission to secure funding Organization and HR plans Staffing key positions with people with right competencies Staff training, Policies & Procedures manual Contingency planning to address priority issues |
| 9. Information Systems – Infrastructure changes required to support SADI project management | Delays in implementing new requirements will impact ability to effectively manage program | High likelihood Moderate impact |
Backup systems to track and monitor projects pending IS implementation |
4.4 Risk Matrix
The following risk matrix positions each of the risks (numbers correspond to risks identified in Section 4.3) with respect to their likelihood and impact.
| Impact | Severe |
Medium Risk | High Risk (3) |
High Risk |
Moderate |
Low Risk (6,7) |
Medium Risk (1,4,8) |
High Risk (2,5,9) |
|
Minor |
Low Risk | Low Risk | Medium Risk | |
| Low | Medium | High | ||
| Likelihood | ||||
Risk management strategies;
Low risk events – occasional monitoring to assess changes in risk
Medium risk events – regular monitoring to assess changes in risk and to initiate management action as required
High risk events – Ongoing monitoring and review by the ITO Risk Management Committee to ensure mitigation strategies are effective and to undertake management actions as appropriate.
4.5 Key Risks
In addition to the risk mitigation strategies, described in Section 4.3, incremental risk management activities will be undertaken for the following key risk events;
- Large Funding Asks – Projects requiring significant financial contribution will be considered for funding from the FISC.
- Due Diligence vs Responsiveness to Applicants – An internal audit of SADI is scheduled for 2008-09 to assess the design and operation of its management control framework and of its compliance to policy.
- Performance Outcomes – Ongoing case studies will be used to track specific project results. These results will help communicate overall program outcomes and benefits.
Monitoring and Evaluation Plan
5.1 Performance Measurement Plan
The table presented in Figure 1 at the end of this section is directly based on the logic model developed for the SADI program. For each outcome in the logic model, the table outlines:
- the performance indicators;
- what method of data collection will be used or the sources of information;
- when the information should be collected; and
- who should be responsible for collecting, analyzing and reporting this information?
In developing the table, the following factors were taken into consideration:
- projects will often take many years to realize their expected outcomes and, as a result, the final or ultimate outcomes may not be measurable during the period associated with this RMAF; and
- in the early years of programming, there will be a very limited number of projects upon which to report; it will therefore be necessary to ensure that the requirements of the Access to Information Act are met; as such, confidential business information may be collected for management purposes but not reported in the early years to ensure confidentiality.
The key performance measurement strategy for the SADI will be to assess progress in terms of its contribution to recipient company and sector innovation leading to competitiveness, increased technological base, innovation, community employment and various public good benefits. These priority performance issues will be addressed on an on-going basis. The performance measurement strategy is to collect and analyze performance information for each outcome.
Information related to the performance of project initiatives will be collected as follows:
- Baseline data collection: ITO will assemble existing baseline information from A&D projects funded under the Technology Partnerships Canada (TPC) program. (This program was the predecessor to SADI).
- Recipient companies' periodic reports: Project-specific information provided by recipient companies' periodic reports will pertain to short term and intermediate outcomes. As per the contribution agreement, recipient companies are required to submit periodic reports to the ITO office (See Progress Reports, Annual Performance Reports and Repayment Reports below). Recipient companies' reporting information will provide a means to monitor overall quality of work, capacities, spending and degree of collaboration undertaken in pursuit of a recipient companies' R&D initiative. Reporting information will also provide a means to monitor progress towards the fulfillment of contractual benefits identified in individual contribution agreements; including specific commitments in regards to repayments to the Minister.
- Progress Reports – On a quarterly and annual basis, recipient companies are required to provide a written progress report that describes the progress in completion of the goals stated for the project and milestones.
- Annual Performance Reports – On an annual basis, recipient companies are required to submit a report identifying previous year's progress and achievement related to public policy benefits. The report will include, but not limited to, annual quantitative information related to the performance areas identified in the performance measurement strategy table below.
- Repayment Reports – On an annual basis, recipient companies are required to submit a report that indicating their gross business revenues and repayments made to the Minister.
- Progress Reports – On a quarterly and annual basis, recipient companies are required to provide a written progress report that describes the progress in completion of the goals stated for the project and milestones.
- Project Monitoring: Project monitoring involves managing projects throughout the two main phases of their life cycle and tracking and reporting technology and economic benefits achieved by the projects.
During the R&D phase, which is expected to last 5 years on average, the company works on its technology research and development, and submits its monthly or quarterly claims. During the benefits phase, on average 15 years, program officers will monitor the company's success and sales, and repayments will be collected as detailed in the contribution agreement. Separate groups of monitoring officers will be responsible for monitoring SADI projects in the R&D versus the benefits phase.
The R&D phase monitoring officer's responsibilities will include:
- ongoing communications with recipient companies, including regular site visits
- ongoing progress reviews, including review and assessment of cash flows on all projects to ensure accurate financial reporting
- detailed examination of progress reports and claims, on a monthly or quarterly basis, to recommend payment based on eligible costs incurred
- ongoing tracking and reporting of project performance relative to the statement of work, and the forecasted benefits performance relative to technology innovation and economic benefits
- the preparation of an annual risk assessment for each project
- vetting Annual Performance Reports
- fulfillment of ongoing corporate and statutory responsibilities, including responding to ATIP requests, ensuring the receipt of lobbyist and contingency fee certifications with claims, addressing audit findings, drafting briefing notes
- amending contribution agreements
- negotiating and performing due diligence on amendments to repayment terms when warranted by changes to a company or other conditions.
The monitoring work in the benefits phase is similar to that described above. However, the focus shifts from tracking R&D performance and benefits and managing claims, to tracking the results and benefits and managing repayments.
Benefit tracking involves evaluating the performance of the project relative to forecasted outcomes described in the logic model. Benefits phase monitoring officers will communicate with companies, review progress reports, make amendments, respond to ATIP requests, etc. Any developments affecting commercialization efforts and/or the financial situation of a recipient company must be identified; when required, repayment forecasts must be adjusted, terms and conditions renegotiated and amendments produced.
Specific responsibilities of benefits monitoring officers include:
- regular communications with recipient companies including annual site visits
- review of financial or sales reports from companies as part of monitoring repayments
- collecting repayments
- updating and vetting forecasts
- preparing annual risk assessments regarding repayments for each project
- vetting Annual Performance Reports
- ongoing corporate and statutory responsibilities including responding to ATIP requests, addressing audit findings, drafting briefing notes
- amending contribution agreements.
On an on-going basis, monitoring officers will use a standardized form (see Annex A) as well as consultations will recipient companies to assess project results for the duration of the project phases.
- Longitudinal Case Studies: While some of the performance information will be readily available from the sources noted above, individual case studies will allow for a more in-depth analysis into specific aspects of performance, in particular the details of how the project impacts the company with respect to enhanced R&D capability and the development and commercialization of innovative products and services. Within the first two years of program implementation, approximately five projects will undergo individual case studies. Additional case studies will be conducted in subsequent years during the term of SADI's program mandate with an average of five additional case studies per year being conducted. All selected projects will be subjected to annual updates.
- Formative Evaluation of the initiative: The SADI's formative evaluation will focus on the relevancy, governance and implementation of the program. It will determine whether adjustments should be made and whether progress toward the achievement of the planned outcomes is occurring. To the extent possible, the formative evaluation will provide evidence on the level of benefits achieved and assess how projects in progress are positioned to contribute to desire results. Performance information available through recipient companies' reports and case studies, and ITO monitoring officers, information systems and records will be assessed for the purpose of the formative evaluation.
- Summative Evaluation of the initiative: The SADI's summative evaluation will focus on the issues related to the program's relevance, success and cost-effectiveness. To the extent possible, the summative evaluation will provide evidence on the level of overall effectiveness the program has in achieving its objectives. Performance information available through recipient companies' reports and case studies along with ITO sources of information will be assessed for the purpose of the summative evaluation.
ITO will be responsible for ensuring that recipient companies report progress in a timely fashion as per the individual contribution agreements. ITO will also be responsible for ensuring the data integrity of its performance measurement systems. On a yearly basis, the ITO will be responsible for reassessing the appropriateness of the performance measurement strategy and for updating it as required.
Recipient companies will be responsible for progress reporting, annual performance update reporting as well as for collaborating for the evaluations (longitudinal case studies, formative and summative) and the audits (internal and recipient companies).
Industry Canada Audit and Evaluation Branch will be responsible for ensuring that the formative and the summative evaluations are conducted as planned and in accordance with the TBS evaluation policy, as well as good evaluation practice.
In view of the risks inherent in R&D projects of a high-technology nature, it is recognized that not all projects will succeed, technically and commercially, and consequently not all contributions will generate the original expected benefits. In order to assess SADI's impact, therefore, performance data collected during the first of program delivery will be utilized in order to provide a basis of comparison upon which to analyze program impact in subsequent years.
Despite the inherent risks that these R&D projects represent, recipient companies are expected to make repayments to the Minister regardless of the extent to which the R&D projects succeed or do not succeed.
| Performance Area | Performance Measures | Data Sources / Collection Methods | Timing / Frequency of Measurement | Responsibility |
|---|---|---|---|---|
| Activities | ||||
| Communicate program availability, eligibility, processes, expected outcomes and benefits | Type of launch activities including;
|
Media tracking Website |
Program launch date Ongoing |
ITO |
| Preparation of Contribution Agreements | # of contribution agreements prepared Application processing timeline |
CMIS CMIS |
Annual | ITO |
| Monitoring of Contribution Agreements | # of claims processed # of monitoring activities completed (progress reports, site visits, annual performance reports) # of repayments processed # of CA Amendments processed |
CMIS CMIS /Project Files CMIS CMIS |
Monthly/Quarterly Ongoing Annual Ongoing |
ITO |
| Outputs | ||||
| Website Development, Proposal Guidelines, Program Information, Recipient company Consultations | Availability of website for stakeholders and beneficiaries to gain information Availability of specific program information including;
|
Website Website |
Program launch date | ITO |
| Execution and Amendment of Contribution Agreements | # of Contribution Agreements executed # of Contribution Agreement amendments executed |
CMIS CMIS |
Ongoing | ITO |
| Short Term Outcomes | ||||
| Awareness of SADI by aerospace, defence, space and security companies | # of enquiries received for pre-applicant support
# of proposals submitted:
|
CMIS CMIS Trend analysis |
Annual | ITO |
| Increased strategic R&D of recipient companies | # of R&D projects by strategic R&D activity Total R&D expenditure by recipient companies Total R&D expenditure by strategic area |
CMIS CMIS CMIS |
Annual | ITO |
| Leveraging of private sector investments in R&D | Total private sector investment leveraged
Total R&D investment by non-approved applicants
|
CMIS / Annual performance report Survey |
Annual | ITO |
| Retained / Increased R&D capacities of recipient companies | # and type of retained/new research personnel
|
Annual performance report Trend analysis Annual performance report |
Annual | ITO |
| Increased collaboration between recipient companies and research institutes, universities, and the private sector | Type of new collaborative partnerships
|
Annual performance report Annual performance report Case studies Annual performance report |
Annual | ITO |
| Intermediate Outcomes | ||||
| Canadian public better informed of program benefits | # of media articles published
|
Media analysis Analysis of website statistics |
Annual | ITO |
| Increased development of innovative products and processes | # of innovative products and services generated # of new products and services adopted in new platforms % of projects achieving planned outcomes # of patent applications, patents issued # of IP licenses issued # of academic articles published # of product and process announcements in industry related business publications # of innovation awards and industry recognition |
Final project report / technical review Final project report / technical review Annual performance report Annual performance report Academic publications Industry / business publications Industry / business publications |
Summative evaluation Summative evaluation Annual Annual Annual Annual Annual |
ITO |
| Increased commercialization of innovative products and processes | Annual sales revenue from innovative products and processes attributed to the R&D project Retained / increased involvement in supply chains |
Annual performance report Annual performance report Case study |
Annual Annual Summative evaluation |
ITO |
| Increased knowledge diffusion among recipient companies and collaborative partners | # of joint patent applications, joint patents issued
# of knowledge transfers and exchanges |
Case study | Summative evaluation | ITO / IC AEB |
| Ultimate Outcomes | ||||
| Increased competitiveness of A&D firms | # of projects achieving commercial success Market segment ranking Growth in A&D sector as a % of manufacturing GDP |
Case study Annual Performance Report Final project report/technical review |
Summative evaluation | ITO / IC AEB |
| Contribute to the achievement of broader technological, economic, environmental and social benefits for Canadians | Increased energy efficiency Conservation / saving of renewable and non-renewable natural resources Increased in production efficiencies / reduction in material usage Improved safety # of high quality personnel positions created / maintained Growth in A&D sector % of overall R&D in Canada Increased national security |
Case study | Summative evaluation | ITO / IC AEB |
5.1.1 Audit Plans
Audits of SADI-supported initiatives have a number of objectives. Audits are intended to ensure that:
- appropriate diligence is exercised with respect to the expenditure of public funds;
- the projects are administered in accordance to the terms and conditions in the project funding agreements;
- the provisions of the Financial Administration Act are respected; and
- the quality and adequacy of information used by SADI to monitor and manage the initiative are relevant and available for decision-making purposes.
SADI will closely monitor and be in regular contact with the recipient companies to reduce the likelihood that certain risks will be realized. It will review the risks on a periodic basis to determine if the planned mitigation strategies are effective or if further action is required.
An internal audit of SADI is scheduled for 2008-09 to allow the program to be implemented but also allow for an internal audit sufficiently early in its life to assess the design and operation of its management control framework and of its compliance to policy. Cost of the audit is estimated at $150,000.
AEB's risk-based audit planning process is carried out in accordance with the OCG's draft Statement of Internal Audit Standards. The process is designed to: provide a systematic basis to compare the risks within elements of an audit universe; establish a comparative ranking of those elements and additional analysis; give an indication of how often and in what depth these elements should be audited; and thereby determine the audit resources that will be needed and how they should be allocated.
The risk factors used include: materiality; risk of fraud; amount of discretion in decision making; requests from senior departmental officials; degree of change in personnel; quality of internal controls; extent of IT reliance; and time and results of the last audit engagement.
The higher the audit risk score, as determined by the audit risk assessment process, the more time and attention are devoted to it. Internal audit performs significant audit work each year in most auditable segments rated "high risk" as a result of its risk assessment process.
The table below provides a summary of the various types of audits that will be performed to monitor risk and identify areas requiring improvement.
| Type of Audit | Audit Objective | Responsibility | Scheduling | Cost |
|---|---|---|---|---|
| Recipient Company Audits |
SADI will prepare a multi-year risk based Audit Plan. This plan will have three types of recipient company audits; Revenues, Cost (including large individual Claims) and Compliance (i.e. Lobbyist and payment of contingency fees) audits. The objective of this plan would be to assess the extent to which the recipient company:
This multi-year recipient company audit plan may be revised on an annual basis to:
|
ITO | As per SADI risk based audit plan | To be determined at the time of audit |
| Internal Audits | Internal Audit objectives are as follows: At the program level, the audit will provide assurance that the department has designed and implemented an effective management control framework:
|
IC AEB | As per AEB's FY 2008-09 risk based audit plan | As per IC's audit plan |
The audit risks factors, set out in the table below, will be utilized to determine the need and timing of a recipient company audit. Projects with the highest audit risk scores will be audited on a priority basis.
| Recipient Company Audit Risk Factors Assessed Annually | |||||
|---|---|---|---|---|---|
| Attribute | Potential Scores for Each Attribute and Associated Descriptions | Score | |||
| 1 | 2 | 3 | 4 | ||
| I. On-site monitoring visit conducted | On-site monitoring conducted and no concerns identified | On-site monitoring conducted and some concerns identified | On-site monitoring conducted and important concerns identified | No on-site monitoring conducted | |
| II. Necessary reports received on a timely basis | All reports received in time | Some reports missing | Important or numerous reports missing | Have not received any reports | |
| III. Accuracy of information provided | No errors found in submitted reports | Errors occasionally found in submitted reports | Errors regularly found in submitted reports | Numerous errors and inconsistencies consistently found in submitted reports | |
| IV. Funds were used for the intended purposes | Detailed reporting is provided that clearly indicates that funding was used for the intended purpose | Some funding usage information missing or not sufficiently detailed | Considerable funding usage information missing or not sufficiently detailed | Financial and activity reports provide insufficient detail to determine if funding was used for intended purpose | |
| V. Recipient company experience with contribution programs | Considerable experience | Some experience | Little experience | No previous experience | |
| VI. Amount of funding recipient company is receiving | Less than $5 million | Between $5 and $10 million | Between $10 and $20 million | Over $20 million | |
| VII. Information about successful projects | Detailed information concerning technical challenges and how they were overcome | Some information concerning technical challenges is missing or not sufficiently detailed | Considerable information concerning technical challenges is missing or not sufficiently detailed | No information concerning technical challenges and how they were overcome | |
| Total Risk Score for this Recipient Company | |||||
5.2 Evaluation Plan
Longitudinal case studies will be utilized to probe more deeply into specific aspects of the program in particular the details of how recipient companies are impacted by their individual R&D projects and to what extent have the projects made an impact beyond the individual firms. Recipient company case studies will be conducted on an on-going basis throughout the entire life cycle of the agreement. It is estimated that approximately five case studies will be initiated within the first two years of program implementation. In the three subsequent years, it is estimated that an additional five case studies per year will be initiated. Updates and analysis of each of the case studies will occur on an annual basis and take one to two months to complete. The cumulative cost for all case studies will be approximately $365,000.
An implementation review will be conducted under the guidance of Audit & Evaluation Branch at the end of the first year of program implementation. The implementation review will address the evolving nature of SADI and its emerging management accountability framework. The review will focus on the implementation of the RMAF / RBAF to ensure that appropriate performance data is being collected to support on-going program management and the planned evaluation. The implementation review will take three to four months to complete and will cost approximately $50,000.
A formative evaluation of SADI will focus on the relevancy, governance and implementation of the program. It will determine whether adjustments should be made and whether progress towards the achievement of the planned outcomes is occurring. To the extent possible, the formative evaluation will provide evidence on the level of benefits achieved and assess how projects in progress are positioned to contribute to desire results. Performance information available through recipient companies' reports and case studies, and ITO monitoring officers, information systems and records will be assessed for the purpose of the formative evaluation. The formative evaluation is expected to be completed at the end of 2009-10 and take three to four months to complete at a cost of approximately $130,000 based on the issues and approaches outlined in the table below.
A summative evaluation of SADI will focus on short term and any longer term outcomes that have been realized in order to address issues related to the program's relevance, success and cost-effectiveness. To the extent possible, the summative evaluation will provide evidence on the level of overall effectiveness the program has in achieving its objectives. The economic impacts on official language minority communities (person-years of employment created and/or maintained SME development, etc.) will be considered during the summative evaluation. Performance information available through recipient companies' reports and case studies along with ITO sources of information will be assessed for the purpose of the summative evaluation. The summative evaluation will be conducted in 2011-12 and is estimated to cost $265,000 based on the issues and approaches outlined in the table below.
As previously stated, a typical R&D project will have, on average, a life cycle that spans 20 years. Since the duration of SADI supported projects extend well beyond the time horizon of program renewal a long-term outcome evaluation will be required in order to demonstrate the extent to which long-term and ultimate outcomes have been achieved. The long-term outcome evaluation will be conducted in 2017-18 at a cost to be determined.
Based on the evaluation plan identified above, Figure 2 on the following pages outlines the basic strategy for longitudinal case studies, implementation review, formative evaluation, summative evaluation and long-term outcome evaluation proposed for SADI. The costs associated with conducting the various evaluation methods identified in Figure 2 will be sourced from the SADI program budget.
| Approach | Description | Cost |
|---|---|---|
| Longitudinal Case Studies (2007-12) | ||
| All data collected on each case study will be updated and analyzed on an annual basis to provide for a more in-depth view into specific aspects of project performance | $365,000 | |
| Total Longitudinal Case Studies Cost | $365,000 | |
| Implementation Review (2008-09) | ||
| Review of documents | This will include a review of all available program and project documents using specialized review programs. | $20,000 |
| Interviews with staff, proponents and others | 15 to 20 in-depth interviews will be completed with staff, proponent organizations and others as appropriate. | $20,000 |
| Integrated analysis and reporting | The information received from all sources will be integrated into a report which will addresses each issue | $10,000 |
| Total Implementation Review Cost | $50,000 | |
| Formative Evaluation (2009-10) | ||
| Review of documents | This will include a review of program and project documents, including Annual Information Updates and multi-year SADI performance measurement reports. | $7,500 |
| Analysis of progress reports / information systems / databases | All progress reports submitted for all projects approved by the time of the evaluation will be reviewed and analyzed to provide direct input into the relevant issues. (See Annex C for more detail.) | $7,500 |
| Interviews with staff, recipient companies, and others | 50 to 60 in-depth interviews will be completed with staff, recipient companies and others as appropriate. | $60,000 |
| Analysis of case studies | This will include a review of the on-going case studies. These cases will involve an in-depth examination of specific projects which will cover a broad spectrum of project types. | $40,000 |
| Integrated analysis and reporting | The information received from all sources will be integrated into a report which addresses each issue | $15,000 |
| Total Formative Evaluation Cost | $130,000 | |
| Summative Evaluation (December 31, 2011) | ||
| Evaluation framework | Before the summative evaluation, the issues, methodologies, sample sizes and data collection tools should be refined / developed to ensure that the strategy proposed in this RMAF is still relevant. | $30,000 |
| Review of documents | This will include a review of program and project documents, including APRs and multi-year SADI performance measurement reports. | $50,000 |
| Analysis of progress reports / information systems / databases | All progress reports submitted for all projects approved by the time of the evaluation will be reviewed and analyzed to provide direct input into the relevant issues. This method will also include the review of information systems and analysis of information in program databases. (See Annex A for more detail.) | $50,000 |
| Interviews with staff and other stakeholders | 40 to 50 in-depth interviews will be completed with staff and others (e.g., Industry Canada representatives, other departments, etc.) as appropriate. | $40,000 |
| Survey of recipient companies | While it is difficult to determine how many recipient companies the program will have had by the time of the summative evaluation, it is expected that a survey of approximately 100-150 recipient companies will be required to provide a representative sample of SADI recipient companies. | $25,000 |
| Analysis of case studies | Between 5 and 10 case studies should be included in the summative evaluation. These cases will involve an in-depth examination of specific projects which will cover a broad spectrum of project types. They should include only completed projects. | $30,000 |
| Integrated analysis and reporting | The information received from all sources will be integrated into a report which addresses each issue | $40,000 |
| Total Summative Evaluation Cost | $265,000 | |
| Long-term Outcome Evaluation (2017-18) | ||
| The long-term outcome evaluation will utilize the same approach as that of the summative evaluation | To be determined | |
5.2.1 Evaluation Issues
The table in Figure 3 displays a progression of evaluation issues, starting with those related to implementation, design and delivery and finishing with summative questions of relevance, success (objective achievement) and cost-effectiveness.
| Issues / Questions | Indicators | Sources / Methods | Timing |
|---|---|---|---|
| Continuous Improvement | |||
| How effective are the governance structure and channels of communication for the management and operations of the SADI? | Opinions of stakeholders Incidence of conflicts / problems Identified best practices Comparison to others |
Consultations with members of the governance structure and staff Document review |
Interim evaluation |
| Are the processes in place effective? Have they succeeded in increasing transparency while improving processing speed by the desired amount? | Processes in place Timeliness of processes Satisfaction with processes Incidence of process-related problems |
Review of documents / process flowcharting Consultations with ITO staff, recipient companies, others |
Interim evaluation |
| What changes could be made to improve the performance and likelihood of success of the SADI? | Opinions of stakeholders and beneficiaries Comparison to others Lessons learned |
Consultations with ITO staff, recipient companies, OGDs, TBS and others | Interim and summative evaluations |
| Performance Measurement Systems | |||
| How effective is the ongoing performance measurement system of the SADI? | Progress made towards implementing performance measurement strategy identified in RMAF
Data quality processes in place Reported ways in which performance information is used Evidence of use in planning and decision making Perceived usefulness by key stakeholders and beneficiaries Cost of maintaining system(s) |
Documents Information systems Consultations with ITO staff and recipient companies |
Implementation review, interim and summative evaluations |
| Program Design and Implementation | |||
| Is the SADI design sound? | Perceived level and observed level of use of consultations in program design
Strengths and weaknesses of program design processes |
Documents Information systems Consultations with ITO staff, Industry Canada representatives and other stakeholders |
Interim evaluation |
| How effectively is the SADI being implemented? | Perceived level and observed level of communications to introduce program
Satisfaction with various aspects of program implementation |
Documents Consultations with ITO staff, Industry Canada representatives, recipient companies |
Interim evaluation |
| Success / progress | |||
| Is the SADI reaching its target groups? | # of users # of users vs. # in target groups (i.e., penetration) Profile of users # of projects funded (vs. # of proposals received) |
Database Secondary data |
Interim and summative evaluations |
| Recognizing that results may take years to occur given the nature of the program, what evidence is there that results have or are likely to occur? | As per the performance measurement strategy
|
Analysis of progress reports and other ongoing performance information
Consultations with ITO staff, recipient companies and others Case studies |
Interim and summative evaluations |
| What has been the quality and level of engagement by all key stakeholders and beneficiaries | Level of cooperation / collaboration (firms, sector, innovation community) | Documents
Specialized survey Consultations with all stakeholders and beneficiaries |
Implementation review, interim evaluation and summative evaluation |
| Has the SADI resulted in any unintended impacts / outcomes, positive or negative? | Incidence of unintended impacts | Consultations with ITO staff, recipient companies, Aerospace Industries Association of Canada (AIAC) and others | Summative evaluation |
| What have been some of the barriers / inhibitors and facilitators to success? | Identified barriers and facilitators | Consultations with ITO staff, recipient companies, AIAC and others | Summative evaluation |
| Relevance | |||
| Does the SADI continue to be consistent with department and government-wide priorities? | Linkage between SADI objectives / mandate and IC priorities | Documents
Consultations with ITO staff and others |
Summative evaluation |
| How effective is the SADI in addressing the needs of its intended audience? | Identification of needs Extent to which program is able to address those needs Feedback on effectiveness of program change in better addressing needs |
Documents
Consultations with ITO staff, recipient companies and others |
Summative evaluation |
| Alternatives, Cost-Effectiveness and Lessons Learned | |||
| Are there alternative service delivery approaches that would increase the cost-effectiveness of the SADI? | Duplication / overlap of program with others Level of effectiveness of program in achieving its intended results:
|
Documents
Analysis of progress reports and other ongoing performance information Consultations with ITO staff, recipient companies and others Case studies Database Data on program costs Cost-benefit and / or cost effectiveness analysis |
Summative evaluation |
| What have been some of the lessons learned regarding the SADI? | Identified lessons learned | Consultations with ITO staff, recipient companies and others | Interim and summative evaluations |
5.3 Reporting Commitments
Reporting commitments are outlined in the following table.
| Nature of Report | Responsibility | Audience | Frequency | Estimate Cost |
|---|---|---|---|---|
| Recipient Company Progress Reporting | Recipient companies of SADI assistance | ITO | As per the contribution agreement | As budgeted in ITO's Operating budget |
| Annual Performance Reports (APRs) | Recipient companies of SADI assistance | ITO | Annual | As budgeted in ITO's Operating budget |
| Initiative Annual Report | ITO Office | DM, Minister, Public | Annual | Included in current reporting costs for SADI |
| Initiative Implementation Review | Audit and Evaluation Branch | ITO, ADM, DM | At start of fiscal year 2008-2009 | As budgeted in the RMAF / RBAF |
| Initiative Formative Evaluation | Audit and Evaluation Branch | ITO, ADM, DM | By end of fiscal year 2009-2010 | As budgeted in the RMAF / RBAF |
| Initiative Summative Evaluation | Audit and Evaluation Branch | ITO, ADM, DM, Minister | By December 31, 2011 | As budgeted in the RMAF / RBAF |
| Internal Audit | Audit and Evaluation Branch | ITO Audit Committee, ITO Mgmt, ADM, DM | As per AEB audit plan | As budgeted by AEB |
| Recipient Company Audit | ITO Office | ITO Audit Committee, ITO Mgmt | At the end of each project | To be determined for each individual project |
Annex A: Project Results Monitoring
Project Results Monitoring
As part of routine project follow-up, SADI plans, on a case by case basis, to follow-up with recipient companies in order to establish the extent to which they have achieved expected technical, company-specific, social and economic results. These expectations will have been established during the application and contribution agreement negotiation process.
A basic template for project follow-up is laid out in Figure A-1 below. In select cases (e.g. above certain contribution assistance value or in cases of high strategic interest) SADI project officers may work with outside experts to establish independent assessments of technological success. A survey of users may also be considered in some cases.
SADI's structured project results monitoring will provide both timely feedback on performance and establish the essential information for case studies – to be used by both formative and summative evaluation studies.
The template outlined in Figure A-1 will be tested and refined as part of the Implementation Review.
| Project Incrementality (Direct SADI Influence) | Proponent / Performance Impacts | Industry Sector / Community Impacts | Economy / Societal Impacts |
|---|---|---|---|
| Involvement
↵ project would not have been done without program assistance ↵ contributed to completing the project more quickly ↵ contributed to completing the project more thoroughly ↵ contributed to project occurring in Canada Major ↵ led project which otherwise would not have been done (initial phase) ↵ played major role in jointly supported project ↵ one of a number of contributions (later phases) Minor ↵ one of a number of contributions to successful completion of project (useful, but not essential) |
Technical results from commercialization of R&D
↵ new or improved product ↵ new or improved process ↵ advancement of knowledge ↵ increased technical capabilities ↵ improved quality control ↵ new skills internally ↵ increased efficiency / improved productivity ↵ technology transfer Infratechnology Results ↵ codes, standards, databases, protocols ↵ acceptance of standards Commercial results ↵ increased sales ↵ increased market share ↵ increased profitability ↵ cost savings Organizational effects ↵ increase in employment ↵ diversification ↵ expansions ↵ strategic alliances / partnerships ↵ achievement awards / recognition |
Technological / Science
↵ production process efficiencies ↵ increased science and technology information ↵ technology changes ↵ infrastructure (e.g., standard scientific and engineering data, industry standards, test protocols, and instrumentation) ↵ training of technological problem solvers whose talents can be applied in many areas (highly qualified personnel) ↵ improved linkages in the R&D community ↵ establishment of quality, performance standards Commercial / Economic ↵ patents ↵ licenses ↵ spin off companies ↵ increased sales ↵ cost savings |
Economic Benefits
↵ reduced consumer costs ↵ increased employment ↵ improved competitiveness Societal Benefits ↵ improved security (defence) ↵ improved quality of life ↵ protection of environment ↵ improved energy efficiency ↵ improved public health and safety ↵ education / awareness ↵ public service efficiency gains (i.e., lowered taxpayer burden) |
This publication is available upon request in accessible formats. Contact:
Multimedia Services Section
Communications and Marketing Branch
Industry Canada
Room 264D, West Tower
235 Queen Street
Ottawa ON K1A 0H5
Tel.: 613-995-8552
Fax: 613-947-7155
Email: multimedia.production@ic.gc.ca
This publication is also available electronically on the World Wide Web in HTML format at the following address: www.ito.gc.ca
Permission to Reproduce
Except as otherwise specifically noted, the information in this publication may be reproduced, in part or in whole and by any means, without charge or further permission from Industry Canada, provided that due diligence is exercised in ensuring the accuracy of the information reproduced; that Industry Canada is identified as the source institution; and that the reproduction is not represented as an official version of the information reproduced, nor as having been made in affiliation with, or with the endorsement of, Industry Canada.
For permission to reproduce the information in this publication for commercial redistribution, please email:
droitdauteur.copyright@tpsgc-pwgsc.gc.ca
Cat. No. Iu4-114/2007E-PDF
ISBN 978-0-662-46229-3
60302
Aussi offert en français sous le titre I Cadre de gestion et de responsabilisation axé sur les résultats (CGRR) et Cadre de vérification axé sur les risques (CVAR): Initiative stratégique pour l'aérospatiale et la défense.
