Summative Evaluation Report - Infrastructure Canada Program First Nations Component - Evaluation Methodology

2.0 Evaluation Methodology

2.1 Purpose and Scope of Evaluation

In accordance with the Treasury Board of Canada Evaluation Policy (2001) and Policy on Transfer Payments (2000), the INFCFN implementation framework calls for a summative evaluation of the Program 12 months following program termination.¹

The purpose of this summative evaluation is to provide decision-makers with information on, and an analysis of, INFCFN outcomes, as well as an assessment of the Program's administrative effectiveness in relation to its objectives, and to promote greater accountability and transparency. As the Program was not renewed and in 2007-2008 the First Nations Infrastructure Fund (FNIF) was established, this evaluation will also make it possible to learn from the work accomplished by the INFCFN and to improve the delivery of the FNIF and other INFC programs.

In particular, we wanted to determine the following:

• whether the Program attained the expected outcomes,
• what effects the Program had on First Nations communities, and
• whether the Program was cost-effective.

2.2 Evaluation Issues

Consistent with the Program's logic model (Appendix F) and the Treasury Board of Canada Evaluation Policy, this summative evaluation addresses a number of issues and research questions, which are set out in the evaluation matrix (Appendix G). Our evaluation focused on the following issues and questions:

Evaluation Issue	Main Questions
Progress/Success	To what extent were outcomes attained? What were the effects on First Nations communities?
Cost-effectiveness	Was the Program delivered efficiently and effectively?

The evaluation team set the information requirements for this summative evaluation in cooperation with INAC staff. The evaluation matrix (Appendix G) presents the questions addressed, as well as indicators, data sources, and data-collection methods. The research for purposes of the evaluation started in the winter of 2007 and was completed in the fall of 2008, in accordance with the timeframe stipulated in the INFCFN implementation framework.

2.3 Methodology

2.3.1 Evaluation Design

We performed qualitative and quantitative analyses of the INFCFN's outputs and efficiency. Specifically, our evaluation team and the consultant:

• reviewed and analyzed INFCFN planning documents, policy documents, websites, and other information,
• developed interview guides and draft surveys based on the evaluation matrix,
• consulted extensively with key informants, and
• reviewed case studies and conducted an in-depth examination of selected projects.

2.3.1.1 Review and analysis of files, documents, and websites

Background documents were reviewed before starting interviews and examining case studies. Our review included the following: the Treasury Board submission; the INFC formative evaluation; INFCFN program documents; INAC Capital Facilities Program's Infrastructure Maintenance Plan; other relevant documents; the SIMSI database; and various websites. Appendix A provides a comprehensive list of the documents and studies that we reviewed for this summative evaluation.

2.3.1.2 Development and pre-testing of interview guides

For each category of respondent, our evaluation team developed key informant interview guides and a case-study interview guide.

The interview guides were developed to obtain the views and perspectives of INFCFN and INAC officials regarding the outputs for, and the administrative efficacy of, the Program. The content of the interview guides was based on the evaluation matrix, and the guides were available in English and French.

Using one of the interview guides, we sought the opinions of participating First Nations about the benefits of the Program to their communities in terms of: environmental, community, and other benefits; other infrastructure priorities for their communities; and any issues related to the application of the INFCFN or its reporting processes.

The case-study interview guide was developed to examine projects from different categories in three regions served by INAC. Using this interview guide, we interviewed First Nations stakeholders regarding benefits resulting from the INFCFN (environmental, community, and economic) and other First Nations priorities. Projects and INAC regions were selected for case studies on the basis of the requirements set out in the evaluation framework as well as criteria such as geographic distribution, available funding, and the number of projects approved. The questions addressed by the case-study interview guide were also based on the evaluation matrix.

2.3.1.3 Consultations with key informants

Consultations were held with the following categories of key informants: program managers at INFC and INAC; communications officers at INFC and INAC; INAC regional directors; the officer in charge of INFC's SIMSI database; individuals in charge of INAC and First Nations management committees; band chiefs; band managers; band councils; and tribal councils. Appendix B provides a comprehensive list of the informants consulted.

2.3.2 Evaluation Implementation

2.3.2.1 Interviews with key informants

Table 2 summarizes the interviews with key informants conducted by the INFC evaluation team and by the consultant.

Table 2: Interviews with Key Informants

Interviewees	Survey Administration Method	Number of Interviews Conducted
Program managers at INFC	In person	1
Program managers at INAC	In person	2
Communications officer at INFC	In person	1
Communications officer at INAC	In person	1
Officer in charge of SIMSI at INFC	In person	1
INAC regional directors	By telephone	7
INAC officers and First Nations management committees	By telephone	7
Subtotal		20
Band chiefs and band managers	By telephone	21
Band councils	By telephone	7
Total		48

Our evaluation team conducted individual and group interviews, in either English or French, with a total of 20 INFC and INAC officials. Most officials had not been in their current positions throughout the duration of the projects in question. Interviews were conducted in person or by telephone with INAC regional directors. Interview guides were developed for each group (program managers at INFC and INAC, communications officers at INFC and INAC, the SIMSI database officer at INFC, and INAC regional directors) and distributed in advance in order to allow the informants to prepare.

The consultant conducted individual and group interviews with INAC key informants, First Nations management committee members, band chiefs and band managers, and members of band councils. For this series of interviews, seven interview guides were prepared and distributed in advance, along with an introductory letter outlining the purpose of the study. The interviews were conducted by telephone and were usually completed within 30 to 40 minutes.

Interviews were held to support the case-study portion of the evaluation. For 15 case studies involving 46 projects, the consultant conducted 50 interviews in person. The consultant reported a high degree of correspondence between the data gathered during the course of the case-study projects and the data on the same projects contained in SIMSI.

2.3.2.2 Case Studies

The consultant carried out 15 case studies in three regions served by the INFCFN: the Atlantic region; Ontario; and British Columbia. These were regions for which we did not have program performance data. For each of these regions, the consultant selected five infrastructure projects (from categories including water, sewers, roads, community buildings, and cultural centres) to be developed as qualitative case studies.

The case studies involved reviewing key documents for each project, conducting site visits, taking photographs of projects, and holding in-person interviews with INAC officials and First Nations members with expertise relating to the type of project. Questionnaires were sent to the informants in advance to allow them to gather needed information.

2.4 Limitations

The main objective of this summative evaluation was to provide decision makers with information concerning the Program's achievement of expected outcomes, effects on First Nations communities, and cost-effectiveness and efficiency.

INFCFN is one of many factors that contributed to the improvement of infrastructure in First Nations communities. A number of other factors, including the existence of complementary programs, the prevailing economic conditions, and the availability of resources, also play a part in building and improving the infrastructures of First Nations communities.

Given this context, it is not easy to determine exactly which outcomes are attributable to the INFCFN and which are attributable to other factors or programs.

Moreover, our findings on the progress/success and cost-effectiveness of the Program are based, for the most part, on the information gathered through our interviews and case studies, as well as on our review of key documents and data related to the Program.

It should be noted that we limited our investigation to 46 chosen projects in three INAC regions. Therefore, the results obtained cannot be generalized to all INFCFN projects. The results were specific to those infrastructure projects that were discussed in case-study interviews. SIMSI data on the case-study projects aside, for most projects, we found that SIMSI data were not up-to-date and included only the information entered at the application stage of the approval process. Consequently, quantified information on outcomes and administrative efficacy were not readily available from SIMSI for this study. Further, these data were not available when requested during site visits to INAC regional offices.

In fact, we found that most INFCFN project managers in First Nations communities did not follow up on output benefits with respect to their infrastructure projects, and were not familiar with SIMSI. With regard to SIMSI as a management tool for INFCFN, we found that the consultations and monitoring carried out by INAC and INFC with respect to use of SIMSI were not extensive enough. As a consequence, in the context of this summative evaluation, in the winter of 2008, nearly one year following termination of the Program, INFC sent a hard-copy print-out of data found in SIMSI to INAC regional offices via INAC headquarters in order to assist them in validating and updating data for each INFCFN project.

As a result, for the infrastructure projects that we assessed, even though the outputs and administrative efficacy were evident, in the absence of SIMSI data, it was not possible to quantify the results. As such, our findings are limited by missing SIMSI data.

As part of the key informant and group interviews, as well as the case-study interviews, we asked interviewees to provide any key documents and to identify the measures of expected outcomes used for their projects. In most cases, these documents were not readily available; as well, few interviewees were able to remember details of projects that took place between 2002 and 2004. In some cases, staff that would have been able to provide documents or data was no longer employed by these First Nations communities.

In preparing for interviews with INFC and INAC officials, it was difficult to obtain most operational documents, and even some of the foundational documents, related to the Program. As a consequence, data comparing the initial infrastructure needs of First Nations communities to the data related to program outcomes were not available. Moreover, for this summative evaluation, we did not look at information from the First Nations that did not submit applications for project funding or First Nations that had submitted applications that were not approved.

^[1]Federal policy on evaluation requires that federal government departments evaluate issues related to relevance, results, and alternatives of their policies, programs, and initiatives in order "to ensure that the government has timely, strategically focused, objective, and evidence-based information on the performance of its policies, programs, and initiatives to produce better results for Canadians."