Summative Evaluation Report - Infrastructure Canada Program First Nations Component - Evaluation Key Findings
3.0 Evaluation Key Findings
The main objective of the First Nations component of the ICP was to improve the quality of life for First Nations communities through investments that: enhance the quality of their environment; promote their long-term economic growth; enhance their communities' infrastructure; and build 21st century infrastructure through the use of new approaches as well as best technologies and practices.
For the purpose of this summative evaluation, our evaluation team measured program success/progress in relation to the achievement of the outputs, immediate outcomes, and intermediate outcomes outlined in the evaluation matrix (Appendix G). It is too early to accurately measure progress/success towards final outcomes of the Program and therefore this will not be covered in this evaluation.
As of December 3, 2007, the total ICP-FN amount disbursed to the First Nations communities to enhance their infrastructure totalled $30,382,690 for 97 projects. This means that INAC regional offices did not spend all the funds allocated for each province, territory, and region, as shown in Table 3. A sum of $742,310 was returned to INAC HQ. According to those we interviewed, the main reason for this was that some First Nations communities were unable to come up with their share of the funding, and therefore could not participate in the Program. In addition, an amount of $334,000 was transferred for the administration of the Program within INAC. Yet, during the course of our evaluation study, we did not find records detailing how this amount was spent.
Table 3 Funds Allocated and Funded for the ICP-FN
|Allocated||Funded||Difference||Diff. as % of Budget|
The ICP-FN contribution share was allocated to the INAC regional offices that entered into contribution agreements with eligible applicants. INAC allocated federal government funds among regional First Nations communities, according to their population and unemployment rate, using the same formula as what was used for ICP national.
In total, INAC regional offices received 263 proposals for infrastructure projects from First Nations communities, and approved and financed 97 projects. Each proposal went through a screening, review, and selection process at the INAC regional office level. As a result, 97 contribution agreements were signed between the Crown and First Nations communities, which led to increased investments to build new infrastructure or to rehabilitate or expand existing infrastructure in First Nations communities. For information on applications received and projects funded, SIMSI data has proven trustworthy, as confirmed by an analytical cross-check with INFC financial project data. As a result, we generated Table 4 from SIMSI data.
Table 4 Number of Projects Funded vs. Number of Applications Received
|Province/Territory||Applications Received||Funded Projects|
|NWT||n/a||n/a (see note)|
Source: SIMSI (Shared Information Management System for Infrastructure). Data for projects in the NWT were not included because the funding allocated for NWT was administered under the terms and conditions of the Northwest Territories federal/territorial agreement. NWT reports on this source of funding outside the scope established by the TB submission for ICP-FN. Therefore, data for the NWT are not included in this table but are included in tables and figures dealing with overall funding in the Program.
As illustrated in the chart below, First Nations communities in British Columbia received the most funding under the Program, followed by First Nations communities in Manitoba, Ontario, Saskatchewan, and Quebec.
For managing program information, INFC developed the national Shared Information Management System (SIMSI) to support the delivery of the ICP-FN by providing on-line project registration, approval, monitoring, and reporting capabilities. In addition to setting up a 1-800 number for assistance, INFC provided training to INAC regional office managers on using SIMSI. INAC agreed to promote and use SIMSI to capture, store, manage, and disseminate information related to infrastructure projects carried out by First Nations communities. However, we found that the information entered in SIMSI did not show whether all projects approved and funded were in fact carried out, as only four INAC regional offices (Manitoba, Alberta, Saskatchewan, and Quebec) validated information related to those infrastructure projects carried out in their jurisdictions; for other jurisdictions, we did not receive this cooperation.
INAC regional office staff complained to INAC headquarters about being required to use SIMSI as a reporting system, mainly because its program requirements included entering data into SIMSI on the basis of performance indicators for the projects approved and financed under the Program. The INAC regional employees who were interviewed found that these requirements made their work more complex and burdensome, and some asked that SIMSI be dropped as a reporting system and that the application and reporting processes be streamlined and simplified. There is no evidence to suggest that this was agreed to by INAC headquarters, though record keeping of performance data by INAC regional offices was generally poor and inconsistent. However, it seems that SIMSI staff was responsive to requests for help and training on the software.
The contribution agreements signed between the Crown and First Nations communities covered the two types of investment priorities established for the Program: green infrastructure projects and local infrastructure projects. The Program required more than half of the projects to pursue, as a first priority, environmental objectives. We found that 64 projects, or 66 percent of the total, fell within this category. These included green infrastructure projects such as those to improve water quality, upgrade wastewater systems, improve solid-waste management, and control flooding. The other 33 projects involved secondary priorities such as cultural, recreational, and tourism facilities (11), local transportation infrastructure (16), and affordable housing (3), or took the form of other INAC infrastructure projects (3).
Our evaluation team found that, in general, the projects carried out through the contribution agreements between the Crown and First Nations communities were aligned with the Program objectives and INFC's priorities. These projects covered the main Program objectives, namely: to improve the environment in First Nations communities (mentioned 147 times in the 97 projects); to upgrade community infrastructures (60 times); to build infrastructures for the 21st century through the use of new approaches as well as best technologies and practices (81 times); and to support long-term economic growth in First Nations communities (78 times).
In terms of the Program's communications objective, all projects that received funding were to be announced publicly through a ministerial visit or a news release. We found that only 68 of the 97 projects approved were advertised.
3.1.2 Immediate and Intermediate Outcomes
As outlined in the evaluation matrix, for this summative evaluation of the ICP-FN, we asked eight questions related to the achievement of immediate and intermediate outcomes:
Achievement of immediate outcomes
- Enhanced existing or new infrastructure,
- Increased investment in new or existing infrastructure,
- Enhanced federal government visibility, and
- Increased knowledge of the role of the federal government in infrastructure.
Achievement of intermediate outcomes
- Improved quality of the environment in First Nations communities,
- Promoted long-term economic growth,
- Improved community infrastructure, and
- Built 21st century infrastructure using new approaches as well as best technologies and practices.
18.104.22.168 Extent to which immediate outcomes have been achieved
Enhanced existing or new infrastructure
Further to its analysis of data from SIMSI and the interviews that it conducted with key informants, our evaluation team concluded that the 97 projects funded under the ICP-FN did result in the existing infrastructure being improved and in new infrastructure being built in First Nations communities. Despite our not having a historic benchmark with which to compare the current data on the success of the Program and the progress achieved, most key informants we interviewed were of the opinion that the projects carried out did put more infrastructure in place or upgraded existing infrastructure.
Data from SIMSI describing projects support our findings from interview observations (see Table 5 below). We found that 57 percent of the projects created new infrastructure while 43 percent improved existing infrastructure in First Nations communities. Of the projects involving the building of new infrastructure, 67 percent were first-priority green infrastructure projects and 16 percent were secondary-priority cultural and recreational facilities projects. Of the projects to enhance existing infrastructure, 64 percent were green infrastructure projects, 5 percent were cultural and recreational facilities projects, and 31 percent were local transportation projects.
Table 5: Nature of Projects Funded through ICP-FN
|Project Type||New Infrastructure||Existing Infrastructure||Total|
|Cultural and recreational facilities||9||2||11|
|Local transportation Infrastructure||3||13||16|
Increased investment in new or existing infrastructure
Our analysis of SIMSI data reveals that program's investment of $30,382,690, together with investments of $63,849,881 from INAC and First Nations partners, generated a total capital investment of $94,232,571 for improving infrastructure within First Nations communities; this amount exceeded the original goal of roughly $93 million. In total, INAC and First Nations communities together provided funding for about 68 percent of total eligible costs, while the program funded First Nations community infrastructure needs at 32 percent of total funding.
Table 6: Amounts Contributed to the ICP-FN by All Partners
|Province/Territory/Region||Amount Contributed by INFC||Amount Contributed by INAC and First Nations||Total|
According to interviewees from First Nations communities, the funds contributed through the Program enabled all communities that completed projects to address some of their infrastructure needs. They found that the Program effectively supplemented existing infrastructure projects in their communities.
Enhanced federal government visibility
The INFC and INAC personnel that we interviewed agreed that the communications activities carried out under the Program did not lead to enhanced visibility for the federal government in First Nations communities and explained that this was because the federal government (and in particular INAC) already has a presence on reserves. Therefore, when an infrastructure project is carried out on a reserve, the community is aware that the federal government has contributed to it. Communities are aware that provincial governments do not contribute to the funding of projects in First Nations communities.
Increased knowledge of the role of the federal government in infrastructure
We expected to find that the ICP-FN also had enhanced First Nations communities' knowledge of the federal government's role in the field of infrastructure by announcing all projects through a ministerial visit or news release, but this was not consistently done. Of the 97 projects approved, we found that only 68 were announced, most of these by means of news releases.
Those we interviewed understood the importance of the federal government's contribution in enhancing the infrastructures in their communities.
22.214.171.124 Extent to which intermediate outcomes have been achieved
The INFC and INAC personnel whom we interviewed observed that the effect of projects on First Nations communities could not be measured, chiefly because performance data had not been input into SIMSI for most INAC regions. In fact, of 7 regions, only 3 (Alberta, Manitoba, and Quebec) have submitted complete data that includes information on the benefits of their projects. However, the consensus among those interviewed, including First Nations, is that the Program has been beneficial to First Nations communities relative to the size of the program.
Improved quality of the environment in First Nations communities
For this summative evaluation of the ICP-FN, we assessed whether the green infrastructure projects improved the quality of the environment of First Nations communities, by bringing about improvements to water and air quality, water and wastewater management, solid-waste management, and efficiency in energy use.
On the basis of the interviews that we conducted with INFC and INAC personnel, we conclude that the Program has enabled First Nations communities to improve the quality of their environment. However, although interviewees observed that the projects have resulted in improved water, wastewater, and solid-waste management, they could not back their findings with data nor determine the extent to which water and air quality had improved.
Interviewees from First Nations communities who participated in interviews and case studies also confirmed that the environmental conditions on their respective reserves have improved as a result of the green infrastructure projects funded through the Program. Although there is minimal quantitative data to support this finding, on the basis of the responses of interviewees and our review of case studies, we conclude that the physical environment has generally improved as a result of green infrastructure projects.
In fact, interviewees identified specific benefits provided by road, water, and sewer projects in terms of improved environmental conditions in First Nations communities. The following are some examples:
- Environmental benefits of road projects
First Nations communities that completed road projects identified some of the resulting environmental benefits. Paving community gravel roads resulted in better control of dust in summer months and in a reduction in, or elimination of, the use of oil and other chemical products to control dust. Such products can pollute rivers and other water bodies or seep into the soil and contaminate groundwater. For these communities, dust control translates into better air quality and dust-free living spaces. For example, prior to the road paving in the community of Chemainus First Nation (British Columbia), there were many health problems resulting from road dust. According to interviewees, since the completion of the paving projects dust levels have dropped considerably. This has led to improved health for many people in the community, particularly those with asthma.
- Environmental benefits of water and sewer projects
Those we interviewed in First Nations communities where water and sewer infrastructure projects were carried out noticed that the improvements to these systems had led to cleaner water supplies. In some cases, nearby lakes and rivers were cleaner as a result of the conversion of residences and buildings from septic systems to a sewer system where wastewater was directed to a treatment plant.
Conversion of septic systems to sewer systems can also improve air quality. Septic systems that become overloaded or that require maintenance can emit unpleasant odours. For example, many homes as well as the band offices in the community of Westbank First Nation (British Columbia) were taken off septic systems and connected to the main sewer system following sewage backups and flooding. Those we interviewed stated that such situations no longer occur.
In the First Nations community of Big Cove (Atlantic), manholes were patched and sewage pumps were upgraded in order to handle the waste flowing through the system. Prior to the upgrades, waste would overflow from the sewer system and pollute the oceanfront. As a result of the upgrades, sewage overflow has been eliminated, and residents of the community can now enjoy the oceanfront without fear of contamination.
We also found that the expansion of water-distribution systems has led to improvements in the environment of First Nations communities. For example, the First Nations community of Akwesasne Mohawk Nation (Quebec) had formerly connected a number of its residents to two water-treatment plants, a situation that created problems related to water pressure and led to unsafe drinking water coming from one of the plants. With the connection to a single main water-treatment plant, operating and maintenance costs have been reduced, and the overall quality of water has increased. Additionally, the community made use of modern techniques for ensuring that its water is of high quality by adding limited amounts of chlorine to the water system.
Our evaluation team found from interviews that some road, waste-management, and landfill projects funded through ICP-FN have had beneficial effects on the general health of First Nations communities that participated in the Program. For instance, in the Whitefish River First Nation community (Ontario), a landfill site was remediated thereby reducing the amount of pollutants entering the environment, according to the interviewee. This has ensured a safer environment in terms of public health and enabled the people of this First Nations community to start a community recycling program to reduce the amount of solid waste going into the landfill.
Promoted long-term economic growth
The INFC and INAC personnel we interviewed expressed the view that most infrastructure projects funded under the Program were not expected to have a significant effect on the long-term economic performance of First Nations communities. There are a few exceptions. For example, some interviewees from First Nations communities where infrastructure projects were carried out to boost tourism stated that tourism had increased as a direct result. In addition, paving roads provided easier access to and within communities, thus saving time for travellers and those transporting goods and people. Paved roads also result in less wear and tear to vehicles, a clear economic benefit to owners of vehicles with the potential to reduce the cost of transportation on reserves.
In the Chippewas of Kettle and Stony Point First Nation community (Ontario), vehicles were often damaged by unpaved or poor roads. The interviewees reported reduced vehicle damage, and thus less money spent on repairs following the paving of these roads.
Other economic benefits derived from projects funded under the Program include full- and part-time jobs to residents of First Nations communities during their construction phase, as well as following project completion for ongoing operations. For example, the construction of the Adams Lake Recreation and Conference Centre (British Columbia) provided five ongoing full-time jobs. As well, the Osoyoos Nk'Mip Desert Cultural Centre was the catalyst for an integrated tourist complex that has created over 100 jobs, according to interviewees.
Other examples of economic benefits for First Nations communities that completed projects funded through the Program include the following:
- A reduced need for bottled water and lower costs for maintaining and emptying septic systems and holding tanks as a result of water and sewer projects
For example, the Woodstock First Nation community (Atlantic) had been experiencing a number of problems relating to water supply and water quality, which a new reservoir and treatment system have alleviated. There is no longer a need for bottled water, as the chlorination system has adequately resolved the contamination issues.
In the First Nations community of Sunchild (Alberta), more than 40 homes and community buildings were taken off septic systems and private wells by building new water and sewer systems and a new water treatment facility. This eliminated the need to truck water to these homes and for bottled water to be used.
- Increased land available for development
In the Madawaska Maliseet First Nation community (Atlantic), storm sewers, water pumping and piping systems, and roads were laid to make room for a new development. The building of this infrastructure allowed for the potential of about 20 new homes, with 5 currently in existence.
Similarly, in the First Nations community of Westbank (British Columbia), the extension of sewers and the introduction of treated water distribution opened up land for new housing lots.
- Increased community safety
For instance, in the First Nations community of Oneida Nation of the Thames (Ontario), the main road in and out of the reserve was laden with potholes and cracks. Consequently, people had difficulty reaching the Band's administration and social services offices. Furthermore, emergency vehicles had to reduce speed in order to proceed along the road, thus jeopardizing their response times. Funding under the ICP-FN allowed for the road to be reconstructed. As a result, traffic on the road has increased, and response times for emergency vehicles have decreased.
- Safer and more efficient movement of people and goods
After building a passenger/freight vessel, three First Nations with no road access to their communities near Prince Rupert (British Columbia) now have a scheduled, reliable ferry service connecting them to Prince Rupert. Residents are now able to schedule medical and dental services, and to shop for provisions, furniture, and other supplies. Before the completion of this project, these First Nations communities were served only by charter aircraft, special chartered barge service, or private transport on fishing vessels. Based on this interview data, we found that the accessibility of these three remote reserves has improved considerably by the availability of this scheduled ferry service, operated in partnership with British Columbia's ferries.
Improved community infrastructure
Our evaluation team found that community infrastructure projects funded under the Program were fewer in number compared to green infrastructure projects, but have a more readily measurable effect on First Nations communities. In fact, after the development of some community infrastructure projects, those individuals we interviewed noticed an increase in the number of cultural and recreational events held in First Nations communities. They also found that the overall health and safety of community residents was improved as a result of some road, waste-management, and landfill infrastructure projects.
In Quebec, for example, an indoor rink built in the First Nations community of Pikogan gave local people access to a new recreational facility, drawing in people from the surrounding area and leading to the organization of hockey tournaments. In Saskatchewan, the development of community centres, for example, in the First Nations community of Birch Narrows Dene Nation, has provided access to much needed meeting space as well as space for cultural activities, organized sports, and after-school activities for residents of the community.
In British Columbia and Saskatchewan, the development of cultural and recreational centres by the Adams Lake First Nation (British Columbia) and the remote community of Buffalo River Dene Nation (Saskatchewan) created much-needed access to meeting places and space for residents of these reserves, as formerly they had limited alternatives. For these two First Nations, the cultural and recreational centres have improved the social and cultural life in their communities, providing a place for physical education and sports, for elderly residents to gather, and for counselling and other community services.
Built 21st century infrastructure using new approaches as well as best technologies and practices
We found that most projects funded under the Program used standard, rather than modern techniques, methods, and approaches to upgrade existing infrastructures or to build new infrastructures in First Nations communities. This is explained by the general lack of means, resources, and expertise with regard to infrastructure projects that characterizes most First Nations communities.
Interviewees also mentioned that, in spite of this fact, in some First Nations communities, some of the projects funded under the Program did follow modern infrastructure construction methods and best practices to rehabilitate or expand existing infrastructure. In general, modern methods were used to repair and replace the existing infrastructure. Interviewees reported that these infrastructure projects had improved road, sewer, water distribution, and waste management systems.
With regard to the efficient use of existing infrastructure, the interviewees in First Nations communities found that some infrastructure projects funded under the Program had led to their communities making more effective use of existing infrastructure or finding additional uses for it. For example, some First Nations communities extended treated water distribution systems to neighboring municipalities. The passenger/freight vessel in service to the First Nations communities near Prince Rupert uses existing docking facilities in Prince Rupert and at the First Nations communities, thus making better and more extensive use of existing infrastructure.
In our evaluation of the cost-effectiveness of the ICP-FN, we assessed whether the most efficient and effective means were used to deliver the Program, and how cost-effective the Program was in achieving expected outcomes.
One indicator of efficiency is the level of administrative costs. As intended, INAC was the lead department in delivering the Program through its Capital Facilities Program. Treasury Board approved funding in the amount of $334,000 for the operational needs of INAC with respect to the management of the ICP-FN. However, detailed data on the funds expended for the administration costs were not available to conduct an analysis of the Program's cost-effectiveness. Given the scarcity of data on the Program's administrative costs, we were unable to determine unequivocally whether the Program was delivered efficiently and effectively. Nor did our review of the Program's documentation, websites, and databases tell us whether the Program had been delivered effectively and efficiently. Our findings on the cost-effectiveness of the Program are based largely on qualitative information.
Some comments made by interviewees from INFC and INAC suggest that the Program was not delivered effectively and efficiently. The initiative added procedures that were different from those in the Capital Facilities Program. Consequently, interviewees observed that the management of the ICP-FN was cumbersome. Further, some of these interviewees stated that the FN component recipient's share of infrastructure projects should have been negotiable.
Some of the interviewees at INFC and INAC also recounted that INAC regional offices lacked the human and financial resources needed to meet the Program requirements. In particular, data on projects approved and financed were to be input into SIMSI as performance indicators. Interviewees found that where they had input data, their work was more complex and burdensome relative to using INAC's data collection system. It should be noted that, as a result, the Program does not hold data to allow for tracking project performance. There were no data available from INAC in lieu of missing data from SIMSI.
The First Nations interviewees, who were also program recipients, commented that they were at ease with the INAC programming and way of operating. They were happy with the complementary manner in which the infrastructure programming was delivered, using INAC personnel and program-delivery systems, reporting, and governance structures. They commented that First Nations communities have established good working relationships with INAC. Also, the partnership approach between INFC, INAC, and First Nations communities was seen by recipients as appropriate. Several First Nations members that we interviewed on site stated that the infrastructure projects carried out could not have proceeded without the Program. They confirmed that, in terms of needed infrastructure, these projects were of the highest priority for their communities.
Similar to comments from INFC and INAC interviewees, the First Nations interviewees found that the Program contributed to cost-effective infrastructures within First Nations communities as it has enabled infrastructure to be built and renovated, including some sustainable and cost-saving infrastructure. In fact, although there is minimal quantitative data to support the contribution of the Program to more cost-effective infrastructure in First Nations communities, there are some exceptions. A number of case studies, interviews, and data from Alberta and Manitoba do provide evidence that a number of projects have resulted in cost savings by reducing the maintenance and replacement costs of some infrastructures.
We found, based on SIMSI data, that in Manitoba (a jurisdiction that has consistently input data into SIMSI), for example, the yearly cost-savings yielded by two local transportation infrastructure projects was $160,000. Moreover, the Program's investments to improve the water distribution system and to convert septic systems to sewer systems in some First Nations communities have reduced the following:
- the need to truck water and the need to purchase bottled water,
- operating and maintenance costs for water distribution systems, and
- the costs of maintaining and emptying septic systems and holding tanks; and
- costs relating to sewage treatment.
Furthermore, in Manitoba, green infrastructure projects are credited with $1,600,000 in yearly cost-savings. Along with these benefits, we found that the Program had led to other cost-savings, such as reduced travel times and reduced costs associated with environmental degradation.
3.3 Emerging Themes and Challenges
3.3.1 Resource Availability and Key Infrastructure Priorities
Despite the ICP-FN's capital investment in First Nations infrastructures and the results that the Program has achieved, further investments are required in order to meet the infrastructure needs of First Nations communities. From the information provided by First Nations interviewees and case studies, our evaluation team has concluded that there is still much work to be done on infrastructure in First Nations communities, including:
- upgrading and repairing water systems,
- rehabilitating, paving, and repairing roadways,
- improving waste-management, landfill, and recycling systems,
- building affordable housing, and
- building community centres and multi-purpose buildings.
In order to meet the growing demand in First Nations communities, these priorities require additional resources. For INFC to deliver the Program as intended, staff shortages at the INAC level also need to be addressed.
3.3.2 Challenges to Program Delivery
Although the partnership with INAC helped INFC better deliver the Program and achieve program objectives and although First Nations recipients were satisfied with the Program's delivery, from our evaluation of interviews and case studies, we have identified a deficiency with respect to the internal capacity of INAC to deliver the Program as intended. INAC attributed its insufficient management capacities to its lack of human and financial resources.
3.3.3 Challenges to Monitoring and Reporting
Despite the existence of SIMSI as a data tracking tool, our interviews and case studies revealed the absence of an operational performance and reporting system to support the management of the Program and to provide information on program and projects results. In fact, we found that some program recipients were unable to provide consultants who visited sites with reports provided to INAC or with any data on tracking project results. Some INAC regional offices were also unable to show us reports provided by program recipients or data to demonstrate the achievement of intended outcomes for infrastructure projects.
This situation is due in part to the fact that many INAC regional managers reported that they had encountered difficulties with SIMSI in validating and updating the information already entered in the system related to their respective ICP projects. However, four provinces did effectively use SIMSI. As a result, projects could not be adequately monitored, and the Program did not have the capacity to measure or demonstrate the achievement of project outcomes or to adequately report on program and projects results.
What did the contribution agreements reveal about performance management and reporting? We are unable to respond to this question because the evaluation team did not receive performance reports or annual audit reports upon asking for them. These documents would have provided independent and objective assurance that appropriate information and monitoring processes and systems were in place to collect relevant, reliable information on the outcomes of the ICP-FN. Moreover, we did not find evidence that INFC requested those reports to INAC during the life cycle of the program from 2001 to 2007.
3.3.4 Challenges to Partnerships Management
If responsibilities for the management and delivery of the ICP-FN and for the achievement of results seem clearly defined in the Governance and Accountability Framework between INFC and INAC, their application was not obvious. We found that the findings from documents, interviews, and case studies are silent about the evidence of solid cooperation between key stakeholders with respect to the Program's implementation and delivery. This situation had had an impact on the management of the partnership between the ICP-FN stakeholders.
Notes: In the Governance and Accountability Framework with respect to the ICP designed by Treasury Board, INAC is amongst those responsible for delivery of the ICP, especially for the delivery and management of the First Nations component. This was to ensure that funding dedicated to First Nations will be spent for the benefit of these communities. However, while the ICP-FN would have a separate administrative process, it would otherwise conform in all aspects to the ICP approach articulated for the overall Program.
- Date modified: