National Summative Evaluation of the Gas Tax Fund and Public Transit Fund - 2.0 Evaluation Issues and Methodologies

2.0 Evaluation Issues and Methodologies

2.1 Context

The GTF and PTF Agreements provide that joint evaluations of the programs shall be carried out no later than March 31, 2009, and address at a minimum the issues related to achievement of the objectives of these Agreements, the use of funding, the effectiveness of the funding approach and the effectiveness of the communications protocol. These joint evaluations must seek the input of eligible recipients or their representatives and the results are to be made public.

Thirteen joint evaluations have been completed, covering all signatories of the Agreements save for Quebec since there was no provision for a joint evaluation under the Canada-Quebec Agreement. These evaluations have received the input of 60 provincial or territorial representatives, and 202 municipal or municipal association representatives, via interviews. Municipal input was also obtained through mail or web-based surveys: 693 survey responses were received and analyzed. In addition, 108 case studies of GTF or PTF projects completed or under way were submitted. For Canada, fourteen different federal representatives participated in these joint evaluations.

Under the Agreements, Canada must complete a national summative evaluation incorporating the results of the joint evaluations no later than June 30, 2009. Following the national evaluation the parties may elect to amend the Agreements, as appropriate.

2.2 Methodologies

This National Summative Evaluation builds on the results of the thirteen joint evaluations and adds an overall examination of results and progress to date. Since the PTF program was only a one-year program, emphasis is on the GTF program particularly in view of the extension of GTF funding to 2014.

The 13 joint evaluations were carried out with a common evaluation matrix¹⁰ that was adapted to the specific circumstances in each jurisdiction: six consultants were tasked with these evaluations. Methodologies varied slightly but were based on four essential components:

• Review of documents, files, evaluations, and other literature;
• Key informant interviews, either individually or in focus groups, and in 10 of 13 evaluations, mail or web-based surveys of eligible recipients;
• Analysis of findings according to key evaluation questions;
• Case studies of both GTF and PTF funded projects either completed or underway.

The joint evaluations offer a broad view of the perceptions and opinions as to the relevance, effectiveness and efficiency of the GTF and PTF programs. They also include recommendations that are in some cases specific to a given jurisdiction, while in others common to many.

The National Summative Evaluation adds quantitative analyses on the use of funds, program management and delivery, and capital spending commitments. A detailed analysis of the GTF Agreements both with Canada and between the provinces, territories, associations and eligible recipients was also undertaken. All projects were considered in the analysis of the use of funds in relation to allocation, project categories, as well as that of expected outcomes.

Given the program design, the data pertaining to allocation to recipients, commitments and expenditures are provided by the signatories in the audited Annual Expenditure Reports. Signatories also provide the list of projects and project categories, as well as expected outcomes with the Annual Expenditure Reports. Realized outcomes are reported in the Outcomes Report to be published by the signatories for their citizens.

The financial data provided by the provinces and territories is audited, so there is no concern on the quality of this information.

Information on projects is provided in a list that is part of the Annual Expenditure Report. The quality of this information is dependent on the quality of the information collected by the provinces, territories and associations that emanate from eligible recipients.

Only UBCM's and Manitoba's eligible recipients enter data in the Gas Tax Fund and Public Transit Fund Reporting System; as is the case in systems where there are multiple users entering data there are some inconsistencies but none that are identified as significant.

The project lists provided by the other jurisdictions come in many formats: PDF scanned versions or image files, PDF files, excel files, and even paper copies. This information is then entered into an excel database and the normal risks involved with manually reentering data are incurred. The project data is analyzed to ensure proper project category and type of project identification: the descriptions provided in the project list are used in this process.

Some of the issues encountered in terms of project data are: the lack of a project identification code which makes it difficult to identify projects that carry over more than one year; inconsistent identification of eligible recipients, sometimes including the type of municipality in the name, other times, only the name, which requires manipulations when comparing the allocation database and the project database; inconsistent reporting of start and end dates; and, limited information provided on other sources of funds or inconsistent reporting of other sources.

There are 2 consequences to this situation:

1. Data analysis requires a lot of manipulation, and in some cases, interpretation of the information provided, such as the project category or sub-category that is reviewed in relation to the project description.
2. Data that are generally reported on can be analyzed. This includes, project categories and sub-categories, GTF and PTF funds received and expended, interest revenues and administration expenditures, and aggregated expected outcomes (Air, GHG emissions, Water) which are core indicators for the National Summative Evaluation.

These issues do not compromise the overall quality of the data supporting this National Summative Evaluation.

GTF project data was analyzed for a sample of jurisdictions including Alberta, British Columbia, Manitoba, Ontario, New Brunswick, Nova Scotia and Prince Edward Island. Together, these jurisdictions account for 68.5% of the total GTF allocation and represent 37.2% of all eligible recipients. The sample analysis covered multi-year projects, analysis of project size in relation to the size of the recipients, and analysis of the size of recipients that had not reported expenses as of March 31, 2008. Quebec and Saskatchewan have reported up until now solely on funds committed, so they were not included in the sample data. Because Newfoundland and Labrador signed-on one year after the other jurisdictions, sample data for this region was not included in this sample analysis. Finally, the territories were not included: a very small number of projects have been undertaken, and in the Yukon, only Integrated Community Sustainability Plans have been produced, with no capital investments to report on as yet.

Data on intergovernmental collaboration, communication activities, and progress toward long-term sustainability planning and reporting on outcomes were also analyzed.

For the PTF, project data was analyzed for all recipients.

No additional interviews were carried out for this National Summative Evaluation. However, Oversight Committee meetings were held where comments on the joint evaluation reports and recommendations were noted and taken into account in this National Summative Evaluation.

2.3 Limitations

Though all reasonable efforts have been undertaken to ascertain the methodologies described were both robust and reliable, there are some limitations to this National Summative Evaluation.

The National Summative Evaluation relies on information from thirteen joint evaluations, the limitations of these include:

1. Though carried out with a common evaluation matrix, there are differences in the joint evaluation methodologies such as the set of questions for the interviews or surveys, which may have lead some aspects of the programs to be discussed in more or less detail in some jurisdictions. However, the extent of these differences does not compromise the overall quality of the information and of the opinions expressed in these reports.
2. Findings related to program management and delivery at the provincial/territorial/association levels are dependent on the program delivery scheme in place which varies across jurisdictions. Conclusions and recommendations may have been specific to the jurisdiction and cannot be generalized.
3. Response rates to the surveys, in particular, varied considerably from one jurisdiction to another, however, the overall trends are very similar across the thirteen reports - while the results cannot be generalized they do provide valuable insight.
4. Case study selection was generally undertaken to reflect a mix of projects and recipients in a range of regional settings. The methodologies varied significantly not only in the selection process but in the content of the studies. It is very difficult to extract significant conclusions from their analysis so that they could be generalized.
5. The PTF program, though included in the joint evaluations, has not been as thoroughly examined as the GTF program. The fact that it was limited to one-year and there were a limited number of recipients (189) compared to the GTF (3,647) may explain, in part, the limited findings on the PTF. Analysis of project data was carried out to compensate for the lack of information provided in the joint evaluations.

As for the data analyzed in the National Summative Evaluation, the main limitation is related to the availability of data.

1. Annual expenditure reports are available only for the first 3 years of the GTF and PTF programs, so that findings on use of funds are limited to the period from fiscal year 2005-2006 to 2007-2008.
2. Newfoundland and Labrador was the last jurisdiction to sign the GTF Agreement so reporting data is limited to only 2 years.
3. From 2005-2006 to 2007-2008, Saskatchewan has reported only on monies committed by eligible recipients. Saskatchewan will be reporting on monies spent in the audited annual expenditure report for fiscal year 2008-2009.
4. Under the Canada-Quebec Agreement, the SOFIL ("Société de financement des infrastructures locales") submits an annual audited financial report to the Quebec National Assembly on funds disbursed to municipalities and eligible recipients for all infrastructure programs including the GTF and the PTF. This report is made available to Canada. To date, Quebec data is limited to funds disbursed to eligible recipients and funds committed by them to GTF projects. There is no data available at this point on funds spent on projects. For PTF funded projects, however, the data provided are for monies spent since in these cases SOFIL reimburses eligible recipients for costs incurred. Quebec municipalities must spend all GTF funds by December 31, 2009. They are required to submit a financial report for all GTF funded projects, including results, by June 30, 2010, and an audited report six months later.
5. Some of the commitments under the Agreements are to be met over the lifetime of the Agreements, such as Capital Investment Plans, Integrated Community Sustainability Plans, and capital spending commitments, or at dates that are later than this National Summative Evaluation, such as Outcomes Reporting. Thus, the National Summative Evaluation can only address progress toward compliance with these requirements and commitments.

Finally, it must be noted that the National Summative Report cannot give full credit to the opinions and recommendations provided in the thirteen joint evaluation reports and to the large number of people who participated in these evaluations. The joint evaluations will play an important role in the future, in the context of the extension of the GTF.

^[10]Appendix 1, Figures 1 and 2, Logic Models, Table 2, Evaluation Matrix