Summative Evaluation Report - Infrastructure Canada Program First Nations Component - Appendix G

Appendix G: Evaluation Matrix For the First Nations Component of the Infrastructure Canada Program

Progress/Success: To what extent have outcomes been attained? What were the effects on First Nations communities?

Outputs

Evaluation Matrix for the first nations component of the Infrastructure Canada Program
Questions Indicators Data Sources Data-Collection Methods
Q1. Has the quality of data regarding project outcomes made it possible to report on the objectives met by the Program?

Data quality and integrity

Outcomes expected/attained

Individual in charge from SIMSI

Manager at INAC SIMSI

Interviews

Data analysis

Q2. Generally speaking, do the infrastructure projects selected meet the terms and conditions previously established?

% of ecological infrastructure projects
% of local infrastructure projects
% of federal-government contribution

SIMSI

INFC / INAC administrative files

Data analysis

Analysis of documentation

Immediate Outcomes

Questions Indicators Data Sources Data-Collection Methods
Q3. Has the Program made it possible to enhance federal-government visibility in First Nations communities?

Number of promotional activities conducted

Visibility of federal- government participation (government of Canada logo, etc.)

Program managers at INFC

Communications at INFC / MAINC

INAC regional directors

Band chiefs/Band managers

websites

Interviews

Analysis of websites

Q4. Has the Program resulted in a common understanding (enhanced knowledge and acceptance) by partners of the federal government's role in the area of infrastructure?

Degree of effectiveness of relations between partners in establishing priority infrastructure initiatives

Ability to surmount potential obstacles by effective communications, risk management, and innovative solutions

Program managers at INFC

INAC regional directors

Management-committee Meeting reports

Meeting reports

Band chiefs/Band managers

Interviews

Analysis of documentation

Intermediate Outcomes

Quality of the environment (in their surroundings) in First Nations communities
Questions Indicators Data Sources Data-Collection Methods
Q5. Has water quality in the communities improved? If so, to what extent, and what is the impact?

Project number and quality (benefits)

Number of households connected to municipal water-distribution system

SIMSI

INAC regional directors

Band chiefs/Band managers

Data analysis

Interviews

Case studies

 

Improvement in water quality/number of water-treatment plants meeting standards

Increased production of potable water

   
Q6. Have the projects made it possible to better manage drinking water and wastewater? If so, to what extent, and what is the impact?

Project number and quality (benefits)

Number of plants equipped with a group meter for measuring drinking-water distribution

Reduction in chemicals (e.g., chlorine) used in treating drinking water or wastewater (tonnes)

Number of service interruptions and incidents (e.g., need to boil water before drinking)

Number of incidents recorded (e.g., rejection of wastewater not meeting standards)

Number of households currently served by a wastewater-treatment system and whose wastewater will be treated more effectively

Number of plants having increased their wastewater-treatment capacity

Number of households connected to municipal wastewater-collection and -treatment systems

SIMSI

INAC regional directors

Band chiefs/Band managers

Data analysis

Interviews

Case studies

Q7. Have the projects made it possible to improve air quality? If so, to what extent, and what is the impact? Reduction in waste incinerated by a given municipality (tonnes/year)

SIMSI

INAC regional directors

Band chiefs/Band managers

Data analysis

Interviews

Q8. Have the projects made it possible to better manage solid waste? If so, to what extent, and what is the impact?

Project number and quality (benefits)

Reduction in volume of solid waste (tonnes/year)

Increase in volume of solid waste recycled or composted (tonnes/ year)

Reduction in methane emissions (tonnes/ year)

SIMSI

INAC regional directors

Band chiefs/Band managers

Data analysis

Interviews

Q9. Have the projects made it possible to more efficiently use energy? If so, to what extent, and what is the impact?

Project number and quality (benefits)

Reduction in use of electricity in buildings and other facilities (kWh/meter3/year)

Reduction in use of heating fuel in buildings (BTU/meter3)

SIMSI

INAC regional directors

Band chiefs/Band managers

Data analysis

Interviews

Case studies

Economy in the long term

Questions Indicators Data Sources Data-Collection Methods
Q10. Is the transportation of people and goods more efficient and safer? If so, to what extent, and what is the impact?

Project number and quality (benefits)

Reduction in number of road accidents

Improved traffic capacity

SIMSI

INAC regional directors

Band chiefs/Band managers

Data analysis

Interviews

Case studies

Q11. Have the projects contributed to the economic growth of the First Nations?

Number of permanent jobs directly created by project

Number of permanent jobs anticipated as a result of project (new companies or industries)

Increased investment in private-sector capital projects or economic activities as a result of project

Short-term jobs created

SIMSI

INAC regional directors

Band chiefs/Band managers

Data analysis

Interviews

Case studies

Q12. How have telecommunications benefited public institutions and remote communities?

Project number and quality (benefits)

Number of users (Internet, telephony, etc)

Number of institutions equipped with telecommunications services (high-speed Internet)

SIMSI

INAC regional directors

Band chiefs/Band managers

Data analysis

Interviews

Case studies

Q13. Have the projects made it possible to stimulate the tourist market? If so, to what extent, and what is the impact?

Project number and quality (benefits)

Number of tourists/visitors in community as a result of project

Increase in average tourist stay in a given community

SIMSI

INAC regional directors

Band chiefs/Band managers

Data analysis

Interviews

Case studies

Community Infrastructure

Questions Indicators Data Sources Data-Collection Methods
Q14. Has the number of affordable housing units increased? If so, what is the impact?

Project number and quality (benefits)

Number of additional affordable housing units

Number of occupants in new affordable housing units

Benefits for occupants of new affordable housing units

SIMSI

INAC regional directors

Band chiefs/Band managers

Data analysis

Interviews

Case studies

Q15.Have the projects made it possible to improve or increase the number of cultural and recreational facilities? If so, to what extent, and what is the impact?

Project number and quality (benefits)

Number of visitors in heritage or cultural facilities

Number of heritage sites designated for preservation and renovation

Projected number of annual users of recreational facilities

SIMSI

INAC regional directors

Band chiefs/Band managers

Data analysis

Interviews

Case studies

Q16.Have the projects made it possible to increase community safety/security? If so, to what extent, and what is the impact?

Project number and quality (benefits)

Decrease in emergency-vehicle response time (number of minutes)

Number of households with reduced risk of flooding

Number of households with better fire protection

Project compliance with building code and safety standards

SIMSI

INAC regional directors

Band chiefs/Band managers

Data analysis

Interviews

Case studies

Infrastructure brought up to 21st-century standards

Questions Indicators Data Sources Data-Collection Methods
Q17. Have the projects made it possible to increase the number of partnerships? If so, to what extent, and what is the impact? Existence of public-private or NGO partnerships/ investments

SIMSI

INAC regional directors

Band chiefs/Band managers

Data analysis

Interviews

Case studies

Q18. Have the projects made it possible to use new approaches and best practices? If so, to what extent?

Confirmation that project was conducted in accordance with best construction practices

Confirmation of reduced consumption as a result of use of demand-management techniques

SIMSI

INAC regional directors

Band chiefs/Band managers

Data analysis

Interviews

Case studies

Q19. Have the projects made it possible to more efficiently use existing infrastructure? If so, to what extent, and what is the impact?

Increased infrastructure use

Reduction in use/operating costs

SIMSI

INAC regional directors

Band chiefs/Band managers

Data analysis

Interviews

Case studies

Q20. Have the projects made it possible to encourage innovation? If so, to what extent, and what is the impact?

Confirmation that project uses best existing technology (i.e., proven and cost effective)

Benefits obtained

SIMSI

INAC regional directors

Band chiefs/Band managers

Data analysis

Interviews

Case studies

Cost effectiveness:

Was the Program delivered effectively and efficiently?
Questions Indicators Data Sources Data-Collection Methods

Q21. (a) Why have funds intended for First Nations component not been used in their entirety ($742,310 returned)?

Q21 (b) Why have funds been used differently than budgeted? (see table 1)

Opinions

Committee minutes

INAC regional directors

Program manager at INFC/INAC

Analysis of documentation
Interviews

Case studies

Q22. Did delivery of the First Nations component within INAC's Capital Facilities Program allow for effective, efficient Program delivery?

Opinions

Benefits associated with incorporating First Nations component of ICP into capital plans

INAC Capital Facilities plan

INAC regional directors

Manager, INAC

Individuals in charge from management committees / member of First Nations

Band chiefs/Band managers

Interviews

Case studies

Q23. Does the Program satisfy First Nations infrastructure priorities? Opinions

Band chiefs/Band managers

INAC regional directors

Individuals in charge from management committees

Interviews
Q24. Does the ICP-FN program complement existing INAC's infrastructure programs? Opinions

Program manager at INAC

Band chiefs/Band managers

INAC regional directors

Individuals in charge from management committees / member of First Nations

Document on infrastructure programs at INAC

Interviews

Analysis of documentation

Q25. Do the infrastructure projects have a maintenance plan ensuring project sustainability? If so, has a funding plan been developed?

Maintenance plan

Funding set aside for maintenance-plan implementation

Opinions on maintenance-plan quality

Opinions on its use

Program manager at INFC/INAC

Band chiefs/Band managers

INAC regional directors

Individuals in charge from management committees / member of First Nations

Meeting reports

Interviews

Analysis of documentation

Q26. To what extent is the Program cost-effective? Are there other ways of attaining the same outcomes? Are these more cost-effective?

Opinions

Program manager at INFC/INAC

Band chiefs/Band managers

INAC regional directors

Individuals in charge from management committees / member of First Nations

SIMSI

Interviews

Analysis of documentation

Q27. What changes could be made to improve Program-delivery effectiveness and efficiency (fund allocation, governance, communications, accountability, stacking, commitments, conditions, selection criteria)?

Changes suggested in keeping with opinions of managers and band chiefs

Program manager at INFC/INAC

Band chiefs/Band managers

INAC regional directors

Individuals in charge from management committees / member of First Nations

Meeting reports

Interviews

Analysis of documentation

Q28. Has the degree of understanding by INAC of community-infrastructure requirements improved?

Community requirements identified by partners

Infrastructure-investment plans

Infrastructure-maintenance plans

Program manager at INFC/INAC

INAC regional directors
Infrastructure-maintenance plans

Infrastructure-investment plans

Interviews

Analysis of documentation

Q29. Has the Program made it possible to improve the decision-making process and to select projects that better satisfy federal objectives and priorities?

More effective decision-making process

Better justification for projects selected

Program manager at INFC/INAC

INAC regional directors

Individuals in charge from management committees / member of First Nations

Meeting reports

Interviews

Analysis of documentation

Case studies

Date modified: