Summative Evaluation Report - Infrastructure Canada Program First Nations Component - Table of Contents and Executive Summary

Infrastructure Canada Program First Nations Component

Final Version May 2010

PDF Version (Size: 374.89 KB) (495 Kb)

Help on accessing alternative formats, such as PDF, PPT and ZIP files, can be obtained in the alternate format help section.

Table of Contents

• Executive Summary
• Background
• Key Findings
  • Progress of the ICP-FN
  • Conclusions
  • Lessons Learned for INAC
    
  • Lessons Learned for INFC and delivery partners
• 1.0 Introduction
  • 1.1 Background
  • 1.2 Program Description
    • 1.2.1 Purpose
    • 1.2.2 Objectives and Scope
    • 1.2.3 Eligible Recipients and Projects
    • 1.2.4 Stakeholders
    • 1.2.5 Planned Results
    • 1.2.6 Program Delivery Model
    • 1.2.7 Program Allocation
• 2.0 Evaluation Methodology
  • 2.1 Purpose and Scope of Evaluation
  • 2.2 Evaluation Issues
  • 2.3 Methodology
    • 2.3.1 Evaluation Design
    • 2.3.2 Evaluation Implementation
  • 2.4 Limitations
• 3.0 Evaluation Key Findings
  • 3.1 Progress/Success
    • 3.1.1 Outputs
    • 3.1.2 Immediate and Intermediate Outcomes
    • 3.1.2.1 Extent to which immediate outcomes have been achieved
    • 3.1.2.2 Extent to which intermediate outcomes have been achieved
  • 3.2 Cost-effectiveness
  • 3.3 Emerging Themes and Challenges
    • 3.3.1 Resource Availability and Key Infrastructure Priorities
    • 3.3.2 Challenges to Program Delivery
    • 3.3.3 Challenges to Monitoring and Reporting
    • 3.3.4 Challenges to Partnerships Management
• 4.0 Conclusions
  • 4.1 Progress/Success
    • 4.1.1 Did the ICP-FN achieve its intended outputs and immediate outcomes?
    • 4.1.2 Did the ICP-FN achieve its intended intermediate outcomes?
    • 4.1.3 Was the ICP-FN delivered efficiently and effectively?
  • Lessons Learned for INAC
  • Lessons Learned for INFC and delivery partners
• Appendix A: List of Documents Reviewed
• Appendix B: List of Interviewees from INFC and INAC
• Appendix C: Interview Guides for INFC and INAC Interviews
• Appendix F: Logic Model for the ICP-FN
• Appendix G: Evaluation Matrix
• Appendix H: List of projects funded by ICP-FN

Executive Summary

The main objective of the Summative Evaluation of the First Nations component of the Infrastructure Canada Program (ICP-FN, or "the Program") is to provide the senior management of Infrastructure Canada (INFC) and the Department of Indian Affairs and Northern Development (INAC) with information concerning the Program's:

• achievement of expected outcomes;
• effects on First Nations communities; and
• cost-effectiveness and efficiency. 

In operation from 2001 to 2007, the Program was created for the purpose of improving the infrastructure of First Nations communities through investments in projects that:

• improve the quality of the environment of First Nations communities;
• promote their long-term economic growth;
• upgrade their infrastructure; and
• build 21^st century infrastructure using new approaches as well as best technologies and practices.

The first priority of the Program was to support projects promoting green infrastructure; its secondary priority was to support projects improving local infrastructure.

Infrastructure Canada's Evaluation Services conducted this summative evaluation of the ICP-FN to address a number of evaluation issues and research questions identified in an evaluation matrix in accordance with the logic model of the Program (Appendices F and H). Our evaluation team, with the help of a consulting firm, conducted the research by:

• reviewing program documents, websites, and databases;
• interviewing key informants; and
• analyzing case studies as well as available financial and administrative data.

This evaluation was challenged by the lack of financial and administrative data and the limited number of program and project files provided by the delivery agency, INAC.

Background

The history of the creation of INFC is important to understanding the role the organization has played in the Infrastructure Canada Program First Nations component (ICP-FN). INFC was created as an independent Office in 2002. Up to that point and since 1994, the Treasury Board Secretariat (TBS) had housed a program called the National Infrastructure Office that provided advice and support to the President of Treasury Board, as the Minister previously responsible for infrastructure. New monies were made available through Budget 2000 that allocated $2.05 billion for new infrastructure. Of this, First Nations infrastructure received $31 million to be administered by the Department of Indian Affairs and Northern Development (INAC). In the intervening two years, between Budget 2000 and the creation of INFC as a separate entity in 2002, the Treasury Board - National Infrastructure Office developed a horizontal Governance and Accountability Framework in 2000 approved by the Prime Minister specifying roles and responsibilities of INFC and delivery partners, including INAC.

As a horizontal initiative, INFC's accountability for the implementation of the ICP-FN was limited. Upon its creation in 2002 as a separate entity, INFC had a mandate for overall policy coordination that it inherited from TBS where the function had previously resided. INFC was in charge of coordinating policy and information to ensure a 'whole of government' perspective by developing frameworks, participating in Federal negotiations, monitoring and reporting on results and similar other tasks including the maintenance of the Shared Information Management System (SIMSI) information management.

The First Nations component of the Infrastructure Canada Program (ICP-FN) was implemented from 2001 to 2007 under the umbrella of the Infrastructure Canada Program (ICP). However INAC was solely accountable for spending and program delivery of the ICP-FN. The Program operated within INAC's Capital Facilities Program, with some adjustment to meet specific requirements of the ICP. Funds for the ICP-FN flowed directly to INAC. Reporting for ICP-FN was the responsibility of the Minister of INAC.

INFC undertook this Summative evaluation in keeping with the requirement in the omnibus TB Submission (2000) for the ICP program as a whole.

Key Findings

Progress of the ICP-FN

Most of the key informants interviewed felt that significant progress has been made towards meeting program objectives and achieving expected results, in terms of outputs and immediate and intermediate outcomes. However, we could not validate this because reliable and comprehensive data was not available from the INFC Shared Information Management System for Infrastructure (SIMSI) or from INAC.

Outputs

We found that the Program was administered as planned through the normal procedures governing INAC's Capital Facilities Program. Of $31,125,000 allocated to the ICP-FN, $30,382,680 was disbursed to fund 97 infrastructure projects through contribution agreements binding the Crown and participating First Nations communities. The allocation of $30,382,680 to the ICP-FN generated a total capital investment of $94,232,571 in improved infrastructure within First Nations communities. This exceeds the Program's initial goal of roughly $93 million.

Of these 97 infrastructure projects, about 66% fell within the Program's first investment priority of green infrastructure projects, and 34% came under the secondary priority of local infrastructure. Most of the key informants we interviewed agreed that all approved infrastructure projects were aligned with both the Program's objectives and INFC's objectives.

Intermediate outcomes

Interviews and case studies revealed that the 97 projects funded under the ICP-FN led to the enhancement of existing infrastructure and the construction of new infrastructure in participating First Nations communities. In addition, the federal government's financial commitment to improve the physical infrastructure of First Nations communities has increased investment in new and existing community infrastructures, by raising additional capital investments from its partners. The First Nations members and staff that we interviewed recognized that, without this federal contribution, their communities could not have carried out their infrastructure projects.

Effects on First Nations communities

The evaluation reviewed the findings of case studies and interviews. It was generally the view of respondents that completed projects have had positive effects on participating First Nations communities, in that they have resulted in improvements in the quality of their environment and infrastructure, better economic opportunities, and the use of modern construction methods and best practices to improve or renew existing infrastructure. The majority of interviewees observed that, in participating First Nations communities, green infrastructure projects have resulted in improved water and wastewater systems and solid waste management. Community infrastructures such as recreational and cultural centres have led to increased opportunities for visits between communities, while local transportation infrastructure has improved road safety on local roads and increased travel in and out of reserves to access products and services.

Given the performance data available for this Program at the time of this evaluation, it is not possible to evaluate the long-term benefits resulting from the ICP-FN. Overall, funding of ICP-FN was modest relative to the needs of First Nations communities. Interviewees agreed, however, that a number of infrastructure projects have been economically beneficial, if not significant. Projects funded under the Program have contributed to cost-effective infrastructure by reducing operating and maintenance costs, allowing for safer and more efficient movement of goods and people in and out of reserves, increased community safety, and improved tourism opportunities. Although First Nations communities largely continued to use their standard techniques, methods, and approaches in building their physical infrastructure, some projects incorporated newer infrastructure construction methods and best practices.

Cost-effectiveness

The individuals interviewed expressed diverging opinions with regard to whether the Program had been delivered cost-effectively and efficiently. While interviewees from INFC and INAC felt that the Program had not been delivered effectively and efficiently as a result of the cumbersome approval process under the Program, the inconsistent use of SIMSI as a monitoring and reporting system, and the financial share required from recipients, interviewees from First Nations communities were generally pleased with the way the Program was delivered. It was found that the Program contributed to making infrastructure projects cost-effective by reducing operating, maintenance, and replacement costs of some infrastructures.

Conclusions

1. Program monies were used as intended. Two-thirds of funds went to the Program's first priority, Green Infrastructure projects.
2. Recipients were generally pleased with the Program.
3. Limited data on results or outcomes were collected regarding the funds invested for most regions; only four of seven INAC regional offices collected and submitted project data results. Indications from interviews with staff and project recipients confirm that projects funded under the Program provided a positive contribution to First Nations communities. Overall, data was not available to evaluate the long-term benefits resulting from the Program.

Lessons Learned for INAC

1. For consistency in program delivery ensure the following:
   • adequate capacity to deliver, and
   • training of staff both at HQ and in the regions.
2. To ensure adequate reporting, implement an auditable and continually improved system for the following:
   • project tracking and monitoring, and
   • collecting and reporting on results with attention to intermediate and long-term outcomes.

Lessons Learned for INFC and delivery partners

1. For complex horizontal initiatives involving multiple federal departments and agencies, partners should prepare an annual performance report on each province, territory, or region they serve in order to gather project information and show program results at the national level.
2. Delivery partners should confirm that cost-effective oversight, internal controls, performance measurement, and reporting systems are in place, at the recipient level, to support the management of transfer payments to recipients.
3. Program objectives should be concrete and clearly defined in relation to program scope and size.
4. A horizontal performance measurement framework and supporting indicators that are realistic, achievable to collect, agreed upon, and congruent with program scope and resources should be developed.
5. INFC should monitor that partners are meeting their reporting obligations including sharing information on performance and results as required by the ICP Governance and Accountability Framework.
6. Identify and share lessons learned and best practices in the delivery of horizontal initiatives with multiple delivery partners.