Summative Evaluation Report: ICP Final Report 2010-10-26 - Executive Summary

Executive Summary

1.0 Introduction

This evaluation, undertaken from April 2009 to November 2009, addresses the Infrastructure Canada Program (ICP), a multi-year time-limited (2000-2001 to 2010-2011) municipal infrastructure program that originates from the 1999 Speech from the Throne. The ICP was a horizontal initiative with a federal contribution of $2.05 billion involving multiple departments and agencies such as Indian and Northern Affairs Canada, Atlantic Canada Opportunities Agency, Canada Economic Development for the Regions of Quebec, Western Economic Diversification, Industry Canada and the Federal Economic Development Agency for Southern Ontario (since 2009).

1.1 Background

The program was designed to draw from the expertise of all three levels of government to develop "cohesive and coordinated approaches to directly address municipal and rural community needs". It was cost-shared by three levels of government and implemented by the federal agencies and departments noted above, in close collaboration with provinces and territories. At the federal level, the ICP was designed to engage federal delivery partners (FDPs) in implementation of the program, so as to build on their expertise and overall knowledge of regional operations, cities and municipal activities.

As a horizontal initiative involving multiple partners, Infrastructure Canada's (INFC) accountability for the implementation of the ICP was limited. Upon its creation in 2002 as a separate entity, INFC had a mandate for overall policy direction that it inherited from Treasury Board of Canada Secretariat (TBS), where the function had previously resided. Responsibility for delivering the ICP resided with the FDPs, INAC and Industry Canada.

The ICP was designed by the Infrastructure - National Office initially located with Treasury Board Secretariat. The ICP was subsequently rolled into INFC, a new department created in August 2002 to provide a focal point for infrastructure issues and programs. These changes affected the management of the ICP with changes in key players responsible for the coordination of the various aspects of the program.

For its part, INFC was responsible for developing the policy parameters (through a Memorandum to Cabinet) and terms and conditions (through TB submissions) for the program information coordination, ensuring a 'whole of government' perspective, leading the federal/ provincial/territorial contribution agreement negotiations, monitoring and reporting on overall results, reviewing recommendations to the Minister of Infrastructure for project approvals (for projects with a federal share of project costs between $1 million and $10 million) and amendments, and similar tasks. Although it participated on management committees and other fora, INFC did not have legal responsibility over program funds expended, the choice of projects, day-to-day oversight of agreements implementation or the performance of the delivering partners.

These roles and responsibilities were set forth in a horizontal Governance and Accountability Framework approved by the Prime Minister at the outset of the initiative (Annex I). INFC was charged with the responsibility for overall coordination including communications and information coordination for performance reporting purposes. Each federal minister responsible for program delivery had financial and parliamentary responsibility and the Minister responsible for INFC was responsible for leading overall policy development and coordination including information management and maintenance of the Shared Information Management System for Infrastructure (SIMSI). INFC was also mandated to undertake this program evaluation. To govern implementation of the initiative, the ICP also had a program administration framework.

This evaluation examined key program evaluation issues including relevance, effectiveness and efficiency, and lessons learned. It was not able to fully assess the cost-effectiveness of FDPs due to the lack of financial and non-financial information. The objective of this evaluation is to provide decision-makers with information and analysis on the results of the program. This evaluation has been conducted pursuant to the Treasury Board Transfer Payment Policy (2008) and the TB Policy on Evaluation (2001), and evaluation commitments arising from Treasury Board Submissions and federal/provincial/territorial agreements. To the extent that information was available, this evaluation also addresses core issues outlined in the Directive contained in the new TB Policy on Evaluation (2009).

2.0 Conclusions

As a horizontal initiative, the program was positioned as a multi-jurisdictional partnership that relied upon the expertise of all three levels of government to develop cohesive and coordinated approaches to directly address municipal and rural community needs. The ICP adopted an innovative model for horizontal management and accountability which demonstrated many best practices and lessons learned on how to manage a government-wide initiative with multiple federal partners.

Several interviewees (INFC, Management Committee, Joint Secretariat and other partners) referred to the ICP as a 'flagship' program in that it laid the groundwork for future federal infrastructure programs. For example, it set out communication protocols and established the joint governance approach (federal-provinces). The working relationship between federal and provincial partners at local level was collaborative, in part because the FDPs had local knowledge and were familiar with their provincial counterparts given they may have dealt with each other prior to the ICP and that relationships of trust and mutual understanding have evolved over time.

Relevance

1. Evidence from in-depth interviews, document review and surveys show that there is a continued need for a program similar to the ICP. The objectives of the ICP were very broad, supporting both economic and environmental objectives. The stated objective of the program was to improve Canadians' quality of life through investments that

  1. enhance the quality of Canada's environment;
  2. support long-term economic growth;
  3. improve community infrastructure; and
  4. build 21st Century infrastructure.

Consistent with the federal sustainable development objective, a minimum of fifty (50) percent of federal expenditures for the overall program was devoted to green municipal infrastructure projects as a first priority. Given the breadth of these objectives, there is continued relevance in 2010 as they are in alignment with the departmental strategic outcomes of "funding for quality, cost-effective public infrastructure that meets the needs of Canadians in a competitive economy, a cleaner environment and liveable communities is provided" and the federal government key priorities as articulated in the Building Canada Plan – a stronger economy, cleaner environment, and better communities.

Success1

2. As of January 31, 2010, the federal contribution of $1.976 billion allocated at the outset of the program generated $7.226 billion in total investments in rural and urban infrastructure. This exceeds the original goal of the program which was to leverage $6 billion in infrastructure investments. Municipalities provided significant levels of matching funding, in addition to the funding provided by the ICP and the provinces/territories. As the program requires contributions from other levels of government, without the ICP funds as leverage, fewer rural and urban new and enhanced infrastructure projects would have occurred.

Evidence collected shows that with federal investment and funds leveraged from other levels of government, 3780 projects applications concerning new and enhanced infrastructure in urban and rural communities have been approved. Projects approved were well aligned with the priorities of the recipients. As such, the ICP has improved the capacities of communities to invest in their priority physical infrastructure needs. Investments devoted to green infrastructure projects and to rural municipal infrastructure projects achieved or exceeded their targets in most provinces and territories. Many of these infrastructure projects have also achieved their intended outcomes. As such, Canadians in participating communities have been provided with better drinking water, improved air quality, improved water and waste water management, and are benefiting from reduced green house gas emissions and improved solid waste management.

Indeed, based on an examination of SIMSI data, survey results and case studies, the program contributed to the enhancement of environmental conditions in some participating communities where green infrastructure were carried out. For example, more than 600,000 households have been provided with improved potable water quality in 411 communities, and 12,000 connected to municipal water service in 80 communities. In some communities, green projects led to a decrease in municipal waste incineration of close to 20,000 tons per year, a decrease of more than 10 billion cubic meters in water lost through leakage in 14 communities, an increase of close to 37 billion cubic meters in solid waste diverted through recycling and composting in 15 communities.

Again, on the basis of SIMSI data and survey results, the program contributed to long-term economic benefits in participating communities, including benefits well aligned with current government priorities, such as job creation, stronger businesses and a stronger tourism industry. As a result of the program, more than 5,000 direct and indirect permanent jobs were created, private sector capital investment increased to $500 million, traffic capacity increased to close 500,000 vehicles per year, and close to 1.2 million tourists visited some participating communities.

Further to environmental and economic benefits, projects funded through the program resulted in the use of the best economically proven technology, construction best practices and new approaches, and more efficient use of existing infrastructure. For example, based on survey results, 75 projects using best available technologies led to numerous benefits including cost savings, improved infrastructure, health and safety benefits, longer life expectancy of infrastructure, environmental benefits and / or improved efficiency while those making more efficient use of existing infrastructure had resulted in reduced operating costs and reduced life cycle costs.

3. The above-noted findings were based in part on SIMSI data. However, such data is incomplete and not consistent across all provinces/territories, resulting in challenges in national performance monitoring and reporting, and issues of data accuracy. SIMSI is an on-line information management tool that enables the management of most Infrastructure Canada programs. Evolving over time, SIMSI permits municipalities to apply on-line for project funding, monitor project status and provide claim information throughout the life of a project. It allows INFC, federal delivery partners, provinces and territories to access funding applications, track and report project approvals, complete and track benefits, financial commitments and expenditures.

Responsibility for ensuring input into SIMSI was jointly shared by the Co-Chairs (federal, provincial) of the management committees in each province. Even as late as five years into the program, federal agencies and departments were still seeking agreement on monitoring and reporting requirements to ensure consistency across jurisdictions and to allow for the analysis of results at the federal level. Better education of all users on what constitutes reliable performance data is required.

4. Despite concerns regarding the accuracy, the reliability and the validity of SIMSI data, there is sufficient evidence that projects achieved intended intermediate outcomes as noted above. Specifically, targeted communities were provided with better drinking water, air quality and water management services, and are benefiting from reduced greenhouse gas emissions and improved solid waste management. However, it is difficult to ascertain without doubt that changes to the quality of life in participating communities is due only to the ICP. While successful implementation of the ICP contributed to an improved quality of life for Canadians, there are many other contributing factors, including for example the existence of complementary programs and economic conditions.

5. Community infrastructure needs funded through the ICP were well aligned with the priorities of most recipient organizations. However, smaller municipalities felt that the application-based approach (as opposed to a direct transfer such as Gas Tax) could be costly and could lead to distortions to their priorities as they were faced with meeting eligibility criteria rather than addressing their needs. Accordingly, flexibility in selection criteria to accommodate smaller communities may be useful. Generally however, the ICP has improved the capacity of the three levels of government to invest in priority physical infrastructure needs of participating urban and rural communities.

6. In addition, the project selection and approval approach associated with the ICP, which required joint decision-making on infrastructure projects by federal, provincial and territorial partners led to an improved level of common understanding of community infrastructure needs by federal and provincial and territorial partners. This collaborative approach allowed community infrastructure needs to be included in discussions on infrastructure issues within the federal government.

7. Moreover, based on evidence collected, the ICP contributed to common awareness, understanding and acceptance by partners of the federal role in infrastructure. According to the interviewees (FDP, Management Committee (MC), Joint Secretariat (JS), municipalities and others), the use of the FDPs was a major advantage in contributing to a common awareness and acceptance by provincial and municipal partners of the federal role in infrastructure. The relationship between federal and provincial partners was largely collaborative, in part because the federal players had local knowledge and familiarity with provincial counterparts. In some jurisdictions, involvement of municipal associations in MC provided a greater sense of partnership and helped to ensure that selected projects met the needs and priorities of the communities. Many recipients that were interviewed felt that the ICP collaborative approach should be considered a best practice.

8. The ICP resulted in increased federal visibility in local communities. However, given the lack of baseline data and subsequent creation of other infrastructure programs, the evaluation cannot determine the extent to which awareness of Canadians of the federal role in infrastructure increased as a result of the ICP alone.

Effectiveness and Efficiency

9. Although the overall design of ICP had many positive attributes, implementation timeframes were severely underestimated at the launch of the program. The design of the program envisioned a program lifespan of 6 years with an end date of March 31, 2006. The initial end date (March 31, 2006) was not sufficient to allow timely project completion as planned in some provinces and territories. Accordingly, the program had to be extended for a further 5 years. This was due to a number of factors, in particular delays incurred in project approvals by both the FDPs and for larger projects, requiring Ministerial approval, in announcements of projects by the Minister, in project amendment processes, and environmental assessments.

10. FDPs exercised due diligence in the delivery of the ICP and complied with all authorities established for the program and related federal policy requirements. FDPs had a high degree of expertise in program delivery and were generally effective in building relationships with provinces and municipalities. However, assessment of the cost-effectiveness of FDPs was not possible due to the lack of financial and non-financial data.

11. Overall, based on the percentage of actual expenditures devoted to green and rural infrastructure, the program targets for green municipal and rural investments were met or exceeded in most provinces and territories.

12. As was the case with the mid-term evaluation of the program, it was not possible to identify which approach in delivery has been most-cost-effective in terms of delivery agency and/or province. This is due to the lack of pertinent financial data to carry out an analysis of the administrative cost-effectiveness, the significant variations in the approaches taken in the implementation of the program, and the difference in levels of completion of the program across jurisdictions. These factors make it difficult to compare administrative costs across jurisdictions.

Given the data available, the evaluators attempted to address this issue by comparing operating budget allocated to each implementing organization to the administrative costs incurred to deliver the program in their respective jurisdictions. Evidence gathered showed that the budget allocated at the outset of the program was not sufficient to cover administrative costs associated to the delivery of the program, except in Ontario. Most delivery partners used their own operating and maintenance budget to subsidize administrative costs incurred for delivering the ICP.

Recommendations

As the Infrastructure Canada Program (ICP) ended in 2010/11, the following recommendations are intended to reflect best practices and lessons learned to inform future Infrastructure Canada (INFC) initiatives.

Success

Recommendation 1:

  • Based on the success of the ICP and where appropriate for future programs that depend on multiple delivery partners, INFC should continue to ensure that departmental objectives and priorities take into consideration provincial, territorial and municipal priorities and needs at both the program design and delivery stages.

Management Response:

  • Agreed. In line with the recommendation and in recognition that infrastructure renewal cannot be addressed by one level of government, INFC will continue to ensure that delivery partners and stakeholders' needs are aligned with departmental objectives in the design and delivery of its programs where appropriate.
  • INFC participated in federal consultations, held in 2006 with provinces and territories, to restore fiscal balance. Infrastructure was identified as a key priority for provincial, territorial and municipal governments. The main components of the Building Canada Plan (BCP), which was established in 2007, including targeted funding and the provision for base funding were devised as a result of these extensive consultations and reflect the needs and priorities of provinces, territories and municipalities.

Recommendation 2:

  • For programs involving multiple jurisdictions, INFC should continue to use joint management committees to promote the involvement of partners.

Management Response:

  • Agreed. INFC will continue to use such mechanisms as joint management committees where appropriate. From the Canadian Strategic Infrastructure Fund to BCP, all INFC programs that have succeeded the ICP used mechanisms including Framework Agreement Committees, Oversight Committees, Agreement Management Committees and Service Level Agreements (SLAs) to support the involvement of partners and stakeholders in the delivery of infrastructure funding.

Recommendation 3:

  • INFC should rely on the resources and knowledge of delivery partners (federal delivery partners (FDPs), provinces) to support the implementation of its programs in the regions.

Management Response:

  • INFC Management is in agreement with this recommendation where FDPs have resources and knowledge that can support optimized program delivery, notably for programs targeting locally significant infrastructure. However, it should be noted that for programs involving projects that are nationally significant (such as Building Canada Fund - Major Infrastructure Component), accountability for project approvals and monitoring should rest with INFC.
  • INFC will continue to ensure optimization of FDPs' resources and knowledge through such mechanisms as joint secretariats established with provincial and territorial counterparts.

Recommendations 4-5-6:

  • INFC should ensure that performance information requirements are clearly reflected in SLAs and Contribution Agreements (CAs). As and that such agreements include clear and concrete performance targets and requirements for reporting on results.
  • To avoid duplication in performance data tracking, collection and reporting, consideration should be given to standardizing data information requirements of both INFC and delivery partners. For example, consideration could be given to reflecting the same performance reporting requirements of FDPs and provinces that are in the program's Performance Measurement Strategy, CAs, Memoranda of Understanding and SLAs.
  • Better education of all stakeholders on what constitutes reliable project-based performance data is required to support the utilisation of the Shared Information Management System for Infrastructure (SIMSI) as a useful performance monitoring and reporting tool. For example, the inclusion of actual as opposed to expected results is important to adequately monitor and report on performance.

Management Response:

  • Agreed. Program Operations recognizes the importance of performance measurement and reporting to ensure accountability and results for Canadians.
  • Since the launch of the BCP, INFC has re-aligned its performance measurement activities and has adopted measures to enhance the evaluation of its programs. Clear expected results were developed for all programs, and performance measurement activities have shifted towards outputs and outcomes which the department can control or influence. Targets were also developed for those outputs and outcomes that are better aligned with the mandate of the department.
  • In keeping with the new Treasury Board Policy on Transfer Payments (2008), INFC will create a Performance Measurement Strategy for any new program.
  • With respect to standardization, performance information data requirements are included in SLAs and CAs. Clear performance indicators to monitor and report to INFC will be developed with the support of INFC Evaluation Directorate.
  • In order to avoid duplication of effort and enhance the usefulness of SIMSI, INFC has also developed guidelines and templates to assist recipients in the development of their Progress Reports. Delivery partners will have access to these reports and will be able to use them to monitor programs performance.

Effectiveness and Efficiency

Recommendation 7:

  • As capacity varies widely, project application processes and timelines should be sufficiently flexible and take into consideration the capacity of the target communities for example, project management expertise, size and geographic location.

Management Response:

  • Agreed. INFC will continue to ensure its programs are adapted to the capacity of recipients. Specific programs and processes are now being tailored to communities in recognition that one size does not fit all.
  • Programs are also targeting various communities such as northern, First Nations and rural communities. In such cases, delivery is adapted to generally smaller project sizes.
  • INFC is implementing a risk-based approach to the delivery of its programs which will better recognize and address the needs of communities with capacity issues.
  • Recently, application processes have been streamlined in order to accelerate infrastructure investments for the BCP. Application documentation requirements (paper and on-line) have been decreased and simplified in order to reduce the burden on communities and ensure timely review and approvals for projects. The results of the evaluation of the Economic Action Plan related initiatives currently being undertaken will help inform future opportunities in this regards.

Recommendation 8:

  • INFC should ensure that sufficient time is allocated for the full cycle of program implementation from project selection to completion. In addition, INFC should work with delivery partners on implementing a risk management strategy in case of slippage in project completion.

Management Response:

  • Agreed. INFC has taken action to address this recommendation in programs established subsequent to the ICP. INFC now allows recipients to start construction and incur eligible costs once projects are approved in principle rather than after the signing of a CA. This has reduced delays between project approval and the beginning of actual construction. Environmental assessment requirements have now been streamlined and are being conducted in a timely manner.
  • INFC has developed monitoring strategies to ensure that projects are completed prior to program deadlines. This strategy engages delivery partners and their knowledge of projects/recipients to better assess risks of slippage and mitigation strategies.

[1] The ICP evaluation was conducted based on the TB Policy on Evaluation (2001). Under this Policy, success was one of the core evaluation issues to be assessed.

Date modified: