National Summative Evaluation of the Gas Tax Fund and Public Transit Fund - GTF and PTF SUMMATIVE EVALUATION MATRIX

<< Previous | Table of Contents | Next >>

GTF and PTF SUMMATIVE EVALUATION MATRIX

Issues Indicators Data Sources Data Collection Methods
Success/Progress: What is the success/Progress achieved to date?
1. What is the level of satisfaction with the GTF and PTF programs for recipients and ultimate recipients within each jurisdiction?
  • Opinions of initial and ultimate recipients
  • Opinions of Oversight Committee members
  • Joint Evaluation Final Reports
  • Other Federal/ Provincial Reports
  • Initial recipients
  • Ultimate recipients
  • Oversight Committee members
  • Document review
  • Joint Evaluation Final Reports analysis
  • Other Federal/ Provincial Reports analysis
  • Key informant interviews
2.
  • a. What have been the results of the GTF and PTF projects so far?
  • b. Are there a good alignment between projects selected, vs. category vs. benefits?
  • c. Was the scope of eligible projects wide enough?
  • # and value of GTF and PTF projects (total cost, % GTF/PTF, % others) by eligible investment categories
  • # and value of projects (total cost, % GTF/PTF, % others) completed as planned by intended program outcomes
  • # Project selected that do not correspond to the category and benefits
  • Opinions of initial and ultimate recipients
  • Municipal leaders/associations identify infrastructure needs that have not been met due to funding and or scope of eligible projects
  • Joint Evaluation Final Reports
  • Other Federal/ Provincial Reports
  • Annual expenditure reports
  • Enterprise Data Warehouse
  • Initial Recipient
  • Ultimate recipients
  • Municipal leaders/ associations
  • Document review
  • Joint Evaluation Final Reports analysis
  • Other Federal/ Provincial Reports analysis
  • Secondary data analysis
  • Key informant interviews
3. Did the GTF and PTF programs result in environmentally sustainable municipal infrastructure?
  • # and value of projects (total cost, % GTF/PTF, % others) by investment category and by environmental outcome
  • # and value of projects (total cost, % GTF/PTF, % others) that did not respect eligible categories (or that do not lead to GTF environmental outcomes)
  • Outcomes collected or as follow:
    • Improvements to water quality
      -Improvements to wastewater collection and treatment
    • Reduced per capita tonnage of solid waste sent to landfill
    • Reduced energy consumption and improved recovery and recycling
    • Reduced emissions and GHG emissions (including reductions due to public transit infrastructure)
    • Improvements to transit infrastructure, efficiency and usage
  • Opinions of initial and ultimate recipients
  • Opinions of INFC Operations
  • Opinions of representatives of community and/or environmental groups
  • Joint Evaluation Final Reports
  • Other Federal/ Provincial Reports
  • Annual expenditure reports
  • Performance measurement reports
  • Enterprise Data Warehouse
  • Initial recipients
  • Ultimate recipients
  • INFC Operations
  • Community / environmental groups
  • Document review
  • Joint Evaluation Final Reports analysis
  • Other Federal/ Provincial Reports analysis
  • Secondary data analysis
  • Key informant interviews
  • Case studies of municipal projects
4. Did GTF and PTF contribute to increased planning capacity of municipalities?
  • Funds spent on capacity building
  • # and type of capacity building projects
  • Opinions of ultimate recipients and municipal associations on the benefit of capacity building funds
  • Joint Evaluation Final Reports
  • Other Federal/ Provincial Reports
  • Annual expenditure reports
  • Enterprise Data Warehouse
  • Ultimate recipients
  • Municipal associations
  • Document review
  • Joint Evaluation Final Reports analysis
  • Other Federal/ Provincial Reports analysis
  • Secondary data analysis
  • Key informant interviews
  • Case studies of municipal projects
5. Have the Gas Tax Funds and Public Transit Funds produced incremental capital spending on municipal infrastructure?
  • % of funds from other sources for GTF or PTF projects compared with the total value of municipal infrastructure spending (through federal GTF and PTF contributions and other contributions from provinces, territories or private industry)
  • Average annual capital spending on municipal infrastructure by initial and ultimate recipient over the period of April 1, 2005 to March 31, 2010 greater than the base amount (provided by GTF and PTF).
  • Annual federal infrastructure spending over period of gas tax agreements
  • Initial recipient did not reduce, eliminate or claw back any municipal infrastructure funding currently being made available to ultimate recipients
  • Opinions of initial and ultimate recipients
  • Opinions of INFC Operations
  • Joint Evaluation Final Reports
  • Other Federal/ Provincial Reports
  • Annual expenditure reports
  • Main estimates
  • Initial recipients
  • Ultimate recipients
  • INFC Operations
  • Document review
  • Joint Evaluation Final Reports analysis
  • Other Federal/ Provincial Reports analysis
  • Key informant interviews
6. Did the GTF and PTF programs contribute to municipal long term sustainability planning?
  • % of ultimate recipients who have an integrated community sustainability plan (ICSP) developed through a public process
  • % of ultimate recipients who expect to complete an integrated community sustainability plan (ICSP) within the next 5 years
  • % of ultimate recipients that use (ICS) planning
  • % of ultimate recipients that have another form of sustainability plan
  • % of ultimate recipients (local governments) that have completed a capital investment plan
  • % of transit plans in place
  • Projects completed contained in sustainability plans (Long term capital investment plans, Master Plans, 20/20, (ICSPs)
  • Recipients report improved long term infrastructure investment decisions (financial leveraging, pooling)
  • % of local governments (by size of municipality and by value of GTF/PTF expenditures) that have adopted and use the accounting rules of the Public Sector Accounting Board
  • Opinions of initial and ultimate recipients
  • Opinions of municipal association representatives
  • Joint Evaluation Final Reports
  • Other Federal/ Provincial Reports
  • Annual expenditure reports
  • ICSPs
  • CSP
  • Initial recipients
  • Ultimate recipients
  • Municipal associations
  • Document review
  • Joint Evaluation Final Reports analysis
  • Other Federal/ Provincial Reports analysis
  • Case studies of municipal projects
  • Key informant interviews
7.
  • a. How do GTF and PTF align with the priorities and objectives of the provinces and municipalities?
  • b. Has the program met the province's objectives (priorities as per the bilateral agreement)?
  • c. Has the program met the municipality objectives?
  • d. Have the optional commitments under GTF chosen by the provinces and territories been met (as per the bilateral agreements), and what were the results/impacts?
  • Comparison of GTF/PTF objectives and Provincial/Municipal objectives/priorities
  • Comparison of provincial objectives and achievements
  • Comparison of municipality objectives and achievements
  • Comparison of commitments made in agreements and those met
  • Opinions of INFC staff
  • Opinions of initial and ultimate recipients
  • Joint Evaluation Final Reports
  • Other Federal/ Provincial Reports
  • Bilateral agreements
  • Annual expenditure reports
  • Annual outcomes reports
  • INFC Operations
  • Initial recipients
  • Ultimate recipients
  • Document review
  • Joint Evaluation Final Reports analysis
  • Other Federal/ Provincial Reports analysis
  • Key informant interviews
8. Did the Gas Tax Fund program foster collaborative approaches across all levels of government?
  • Types of support offered by INFC to initial recipients and initial recipients to ultimate recipients
  • # and type of issues raised and resolved between federal, provincial, territorial and municipal stakeholders
  • Time taken to resolve issues
  • New collaborative approaches or mechanisms introduced
  • Opinions of INFC staff, initial recipients and ultimate recipients
  • # of inter-municipal and municipal- First Nations reserve collaborations on Eligible Projects.
  • Opinions of local government representatives
  • Joint Evaluation Final Reports
  • Other Federal/ Provincial Reports
  • Agreements between INFC and initial recipients
  • Agreements between initial and ultimate recipients
  • Oversight Committee meeting minutes
  • Correspondence
  • INFC and initial recipient meeting minutes
  • INFC Operations
  • INFC Policy
  • Initial recipients
  • Ultimate recipients
  • Local government representatives
  • Document review
  • Joint Evaluation Final Reports analysis
  • Other Federal/ Provincial Reports analysis
  • Key informant interviews
  • Case studies of municipal projects
9. What other effects, both beneficial and adverse, have the GTF and PTF programs had on recipients of funding?  (e.g., innovation, capacity to meet requirements of the programs, managing expectations, obtaining support from INFC)
  • i. Are some recipients of funding affected more by the program than others?
  • ii. Are there some effects due to limited capacities?
  • Description of other, unanticipated effects or outcomes in reports on projects
  • Opinions of initial, ultimate recipients and municipal associations on the effect GTF and PTF funded projects have had on both initial recipients (P/Ts) and ultimate recipients (municipalities)
  • Opinions of INFC Operations
  • Opinions of representatives of community and/or environmental groups
  • Number and value of projects undertaken sooner because of GTF/PTF
  • Joint Evaluation Final Reports
  • Other Federal/ Provincial Reports
  • Periodic outcomes reports (performance measurement reports)
  • Specialized reports by municipalities
  • Initial recipients
  • Ultimate recipients
  • Municipal associations
  • INFC Operations
  • Community / environmental groups
  • Document review
  • Joint Evaluation Final Reports analysis
  • Other Federal/ Provincial Reports analysis
  • Key informant interviews
  • Case studies of municipal and transit projects
Cost Effectiveness: Are the most appropriate and efficient means being used to achieve objectives?
10. Have ways of determining the cost-effectiveness of the projects in the GTF/ PTF programs been identified? If not, why.

Is it possible to identify a way to do it at the project/program level?
If yes, how cost effective are the GTF and PTF programs, in terms of:

  • a. Program management and control?
  • b. Funding allocation or delivery mechanisms?
  • Feasibility of reducing program costs
  • Total cost (Capital cost + Administration costs) (from federal, initial, and  ultimate recipients) by unit of benefits
  • Cost of delivery of GTF/PTF compared to other federal (provincial/ territorial) infrastructure programs, transfer payments or grants
  • Cost of program vs. leveraging impact
  • Cost of collaboration aspect of program
  • # and % of municipalities entitled to funding that did not receive funding
  • Allocation of funds versus funds disbursed
  • Inefficiencies noted and recommendations for change made by Oversight Committees/ Initial recipients/ ultimate recipients
  • Opinions of initial and ultimate recipients and municipal associations
  • Opinions of INFC Operations and Policy
  • Joint Evaluation Final Reports
  • Other Federal/ Provincial Reports
  • Annual expenditure reports
  • Oversight Committee meeting minutes
  • Specialized financial studies of selected projects by INFC or municipalities
  • Reports on other infrastructure programs
  • Initial recipients
  • Ultimate recipients
  • Municipal associations
  • INFC Operations and Policy
  • Document review
  • Joint Evaluation Final Reports analysis
  • Other Federal/ Provincial Reports analysis
  • Secondary data analysis
  • Key informant interviews
  • Case studies of municipal and transit projects
11. What changes could be made to improve the performance and success for GTF/PTF? INFC, initial and ultimate recipients, and municipal associations recommend changes to improve the performance and success.
  • Joint Evaluation Final Reports
  • Other Federal/ Provincial Reports
  • Initial recipients
  • Ultimate recipients
  • Municipal associations
  • INFC Operations and Policy
  • Document review
  • Joint Evaluation Final Reports analysis
  • Other Federal/ Provincial Reports analysis
  • Key informant interviews
  • Case studies of municipal and transit projects
<< Previous | Table of Contents | Next >>
Date modified: