Evaluations - Evaluation of Infrastructure Canada's Economic Action Plan Initiatives - Annex 5
Annex 5 – Data Collection Instruments
Interviews
Note: The interview master list is presented to give the reader an overview of the main interview questions used for all EAP interviewees. For program/recipient-specific interview guides, please contact the department.
Question | Indicator | Group | ||||||||||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Senior Managers | Program Staff | Delivery Partners | Ultimate Recipients | |||||||||||||||||||||||||||||||||
Background | ||||||||||||||||||||||||||||||||||||
1. |
[ISF, CC] To begin, please briefly describe your involvement with [ISF or BCF–CC or Top-Up] and how long you have been involved. Are you/ were you involved in the delivery of other INFC programs? If so, which ones. [NRT] To begin, please briefly describe your involvement with National Trails funding and how long you have been involved. |
– | • | • | • | • | ||||||||||||||||||||||||||||||
Initiative Objectives | ||||||||||||||||||||||||||||||||||||
2. |
a) What in your view were the main objectives of the EAP initiative that you were involved in? b) Were the main objectives clear to you? c) In your view, are these objectives being met? If not what were the reasons for not meeting the objectives? |
1.02 6.00 (performance question) |
• | • | ||||||||||||||||||||||||||||||||
Project Approvals, Risk Management and Decision-making | ||||||||||||||||||||||||||||||||||||
Note: Questions in this section apply only to the process related to the initial March 31 deadline for this initiative and do not refer to the process related to project extensions under the EAP extension to October 2011. | ||||||||||||||||||||||||||||||||||||
3. |
a) How were project eligibility criteria communicated? b) Was the communication clear and timely? c) What worked well and what needed improvement? |
2.08 | • | • | • | • | ||||||||||||||||||||||||||||||
4. |
Table 1 (provided separately) provides an overview of the project review and decision-making process for [ISF or BCF–CC Top-Up or NRT]. In reference to Table 1: a) This process was implemented with a view to balancing the need to expedite funding with the need to respect authorities and accountabilities. How effective was this process in achieving this balance?
Please provide examples to explain your response. b) Was the approval process effective? What were the efficiencies or bottlenecks in processing project application (e.g. due diligence, capacity, etc.)? c) Did the size, level, composition and operation of your main decision making bodies support effective decision-making? (SENIOR MANAGEMENT, STAFF) d) What was your level of satisfaction with the quality of information available for decision-making and oversight (e.g. recommendations and supporting information)? (SENIOR MANAGEMENT)
Please provide examples to explain your response. e) What was your level of satisfaction with the information available to assess and recommend projects (e.g. project application forms, certificates of construction-readiness)? (PROGRAM STAFF)
Please provide examples to explain your response. |
2.03 | • | • | • | |||||||||||||||||||||||||||||||
5. |
[ISF, CC] a) Was project assessment for approval done at the appropriate level within INFC? Please explain. [ISF] In some cases, an increase in spending authorities to the Minister meant that ISF projects were approved by the Minister and not by the Treasury Board. This was implemented in order to accelerate the approval process. b) What effect did increased spending authorities have on the timeliness of project approvals for the process related to the initial March 31st deadline? c) What effect did increased spending authorities have on the risk tolerance level at INFC in general? d) Were decisions to approve ISF projects made at the appropriate level? Why or why not? |
2.05 3.02 |
• | • | ||||||||||||||||||||||||||||||||
6. |
Table 2 (provided separately) provides an overview of the tools and mechanisms employed to manage risk for [ISF or BCF–CC Top-Up or NRT]. a) How were these tools and mechanisms used? b) To what extent was risk information used as an input to decision-making? c) Are there other tools, aside from the ones mentioned in Table 2, which were used to manage risk? d) How useful and effective were these tools and mechanisms in managing risk given the requirement for accelerated approvals? What worked well and what needed improvement? |
2.04 |
• | • |
• [not for ISF] [CC only] |
|||||||||||||||||||||||||||||||
7. |
Table 3 (provided separately) provides an overview of the requirements for [ISF or BCF–CC Top-Up or NRT] project tracking. a) How useful and effective were the data collection/reporting tools? (e.g. SIMSI, others) b) Were there any challenges with respect to data collection and reporting requirements? c) How was performance information used (to manage, to report)? d) How useful and effective were these methods in project tracking taking into consideration the requirement for accelerated delivery? What worked well and what needed improvement? |
2.04 |
• | • |
• [FDPs all] [provinces a) and b) only] |
|||||||||||||||||||||||||||||||
8. |
[ISF only] a) Are you aware of any key reviews and audit observations for ISF? b) Were you aware of any management action made in response to these observations? c) Did the management actions respond adequately to the issues identified? d) Were they implemented in a timely manner? |
2.06 |
• | • |
|
|||||||||||||||||||||||||||||||
Partnerships and Stakeholder Engagement | ||||||||||||||||||||||||||||||||||||
9. |
[ISF, CC] a) Do you believe the roles and responsibilities between INFC and delivery partners were well defined? b) Were there any overlaps between your roles and responsibilities and INFC or other partners? c) What assisted/hindered you in understanding your role? (e.g. MOUs, contributions agreements, service level agreement, communication) d) What worked well? What needed improvement? [NRT] a) Do you believe the roles and responsibilities between INFC and NTC organization were well defined? b) Were there any overlaps between your role and that of INFC? c) What assisted/hindered you in understanding your role? (e.g. MOUs, contributions agreements, communication) d) What worked well? What needed improvement? |
2.02 |
• | • | • |
|||||||||||||||||||||||||||||||
10. |
a) Please describe any changes to the way INFC and delivery partners worked together, in reference to projects funded under the initial March 31st deadline, to enable expedited funding [ISF or BCF–CC Top-Up or NRT]? How were these changes implemented? b) Using the scale below, please indicate your level of satisfaction with how these changes were implemented.
Please explain your response. What worked well and what needed improvement? |
2.10 | • | • | • | |||||||||||||||||||||||||||||||
11. |
Using the scale below, please indicate your level of satisfaction with the following:
(NB: Need to separately define a) communication between INFC and delivery partners b) communication between delivery partners and ultimate recipients c) level of cooperation between INFC and delivery partners d) level of cooperation between delivery partners and ultimate recipients e) the division of roles and responsibilities between INFC and delivery partners f) the division of roles and responsibilities between INFC and ultimate recipients (direct relationship) (Not NRT) g) project approval processes for the initial March 31st deadline i) project reporting requirements j) funding mechanisms (i.e. memoranda of understanding, contribution agreements) k) overall service and support provided by INFC Please provide examples to support your opinions. What worked well and what needed to be improved? |
2.10 | • | • | • | |||||||||||||||||||||||||||||||
12. |
a) Did your organization have appropriate capacity to respond to the accelerated delivery that the initiative demanded (e.g. number of staff, office space and equipment, skills/expertise) [ISF or BCF–CC Top-Up or NRT]? b) If not, what were the challenges and how were they overcome? c) What challenges remain? |
2.11 |
• |
• | ||||||||||||||||||||||||||||||||
Management of Organizational Change | ||||||||||||||||||||||||||||||||||||
13. |
a) What measures were taken by management to identify and implement organizational changes (e.g. new tools, HR, organizational structure, process change) at INFC to support rapid implementation of [ISF or BCF–CC Top-Up or NRT]? b) What worked well and what needed improvement? (senior managers and staff) c) Were any additional changes implemented to accommodate the EAP extension to October 2011? How well did this work? |
3.01 |
• | • |
|
|||||||||||||||||||||||||||||||
14. |
a) In general, was INFC adequately staffed with skilled people in the right positions to support implementation of the initiative? b) If not, how was necessary capacity built to overcome any barriers? (e.g. recruitment, training, tools, internal communications, etc.)? c) What worked well and what needed improvement? |
3.03 |
• | • |
• [a) and c) only] |
|||||||||||||||||||||||||||||||
15. |
Using the scale below, please indicate your level of agreement with the following statements for the process related to the initial March 31st deadline.
a) I was part of the solution in delivering this rapid initiative. b) I was engaged and informed about the organizational changes that were brought about to accommodate the requirement for accelerated delivery c) I received clear direction in order to deliver on the initiative d) My workload was manageable e) The division of responsibilities was appropriate and effective f) I was comfortable in my new role in delivering the initiative g) I was comfortable making risk-based recommendations h) I had the right skill set to perform my duties for [ISF or BCF–CC Top-Up or NRT] i) I received adequate training to perform my duties Please provide examples to support your opinions. What worked well and what needed to be improved? |
3.04 | • | |||||||||||||||||||||||||||||||||
16. |
Using the scale below, please indicate your level of agreement with the following statements for the process related to the initial March 31st deadline.
a) Staff was part of the solution in delivering this rapid initiative. b) Staff was engaged and informed about the organizational changes that were brought about to accommodate the requirement for accelerated delivery c) Clear direction was provided to staff in order to deliver on the initiative d) The workload of my staff was manageable e) The division of responsibilities among staff was appropriate and effective f) I was comfortable in my role in delivering the initiative g) I was comfortable making risk-based decisions h) Staff had the right skill set to perform their duties for [ISF or BCF–CC Top-Up or NRT] i) Staff was given adequate training to perform their duties Please provide examples to support your opinions. What worked well and what needed to be improved? |
3.04 | • | |||||||||||||||||||||||||||||||||
Effectiveness | ||||||||||||||||||||||||||||||||||||
17. |
[ISF & CC only] a) Did you apply for an extension to the original March 31, 2011 deadline? b) What circumstances contributed to the need to extend the project? c) Was the extension process effective (e.g. efficiencies or bottlenecks in the extension process)? Please describe how or why it was or wasn't effective. |
6.04 2.03 |
|
• | ||||||||||||||||||||||||||||||||
18. |
Table 4 (provided separately) provides an overview of the delivery models for the initiative. a) What advantages or disadvantages did each of these delivery models provide in relation to the requirement for rapid implementation? (e.g. were there bottlenecks, how could they have been unblocked, what factors facilitated or impeded delivery) Can you provide examples to support your opinion? b) Are there features of these delivery models or other delivery models that should be included in future planning of similar initiatives as good practices? |
8.01 |
• | • | • |
|||||||||||||||||||||||||||||||
19. | How could the initiative be improved or delivered differently by partners (federal, provinces, territories, and municipalities) to better satisfy the need for quick approval and delivery of funds? |
8.02 |
• | • | • |
• | ||||||||||||||||||||||||||||||
20. | Are there any examples of challenges or unintended impacts as a result of the initiative? |
7.01 |
• | • | • |
• | ||||||||||||||||||||||||||||||
21. | Is there anything else you wish to add? |
– |
• | • | • |
• |
Surveys
Note: The survey master list is presented to give the reader an overview of the main survey questions used for all EAP respondents. For program/recipient-specific survey guides, please contact the department.
Project Identification
1. Our records indicate that your organization has received funding for project(s) through the [ISF/BCF–CC Top-Up/NRT] under Canada's Economic Action Plan. (drop down menu)
2. Please, identify up to three of the highest dollar value [ISF/BCF–CC Top-Up/NRT]-funded projects which you are responsible for managing or administering. (Is there any common identifier that we can use to facilitate cross referencing)
3. When you think about the [ISF/BCF–CC Top-Up/NRT]-funded projects for which you are responsible, please indicate:
- How many projects are for new infrastructure? (pick list of numbers)
- How many projects are for the rehabilitation or improvement of existing infrastructure? (pick list of numbers)
4. Which categories best describes the [ISF/BCF–CC Top-Up/NRT]-funded project(s) for which you are responsible? (select all that apply)
[Categories will be separated based on each program's eligibility list]
5. Did your organization also have unsuccessful applications for funding?
6. How would you categorize the type of organization for which you work: (select only one)
7. In which province/territory are/were your [ISF/BCF–CC Top-Up/NRT]-funded projects located? (select one)
Include the following only for ISF/NRT
8. Please identify the type of agreement you entered into to obtain funding from the [ISF/BCF–CC Top-Up]? Not for NRT
Stakeholder Relationships
9. Please complete the questions below regarding stakeholder relationships by indicating your level of agreement with the following statements:
Strongly Disagree | Disagree | Agree | Strongly Agree | Not Applicable | Do Not Know | |
---|---|---|---|---|---|---|
9.01(2.10e) The roles and responsibilities of INFC were clear to me [ISF only]. | ||||||
9.02 (2.10e) The roles and responsibilities of the provinces/territories were clear to me [ISF, BCF–CC only].
9.02 (2.10e) The roles and responsibilities of the NTC were clear to me [NRT only]. |
||||||
9.02(2.10a) I am satisfied with the level of advice and support I received from the province or territory with which I had an agreement. [ISF, BCF–CC only]
9.02(2.10a) I am satisfied with the level of advice and support I received from NTC with which I had an agreement. [NRT only]. |
||||||
9.03 (2.10a) I am satisfied with the level of advice and support I received from INFC (ISF direct agreements only) [ISF] | ||||||
9.05 (2.10) How could stakeholder relationships be improved? |
Project Application and Approval
10. Please complete the questions below by indicating your level of agreement with various aspects of project application, approval and reporting [not Quebec]:
Strongly Disagree | Disagree | Agree | Strongly Agree | Not Applicable | Do Not Know | |
---|---|---|---|---|---|---|
10.01 (2.08) Availability of information on project eligibility was timely. | ||||||
10.02 (2.08) Information on project eligibility was clear. | ||||||
10.03 (2.08) I understood which projects would be eligible for funding. | ||||||
10.04 (2.10b) Project approval decisions were communicated in a timely manner. | ||||||
10.05 (2.10b) Project approval decisions were clear. | ||||||
10.06 (2.10b) I understood why my projects were accepted. | ||||||
10.07 (2.10b) I understood why my projects were rejected. | ||||||
10.08 (2.10b) The project reporting requirements were clear. | ||||||
10.09 (2.10b) I understood my role and responsibilities related to reporting. | ||||||
10.10 (2.10b) Reporting requirements were not overly burdensome. | ||||||
10.11 (2.10b) Overall, I am satisfied with the project approval process. | ||||||
10.12 (2.10b) Overall, I am satisfied with the project reporting requirements. | ||||||
10.13 How could the project application and approval process be improved? |
Project Implementation
11. Are all you projects completed?
12. (2.09, 2.10, 2.11) Did you experience delays or do you anticipate delays in the implementation of your project(s) such that project(s) can not be completed by the original March 31, 2011 deadline?
(skip to Q15)
13. When do you expect your project(s) to be completed?
14. What are the main reasons that the project(s) were/will be delayed? (select all that apply)
15. (2.11) To what extent did the [ISF/BCF–CC Top-Up/NRT] initiative, with its short time frames, affect you?
Significant Negative Impact |
Negative Impact | Positive Impact | Significant Positive Impact |
Don't Know | Not Applicable |
---|---|---|---|---|---|
Go to Q16 |
Go to Q16 | Go to Q17 | Go to Q17 | Go to Q18 | Go to Q18 |
16. (2.11) If you identified any negative impact, please identify the issues. (select all that apply)
17. If you identified a positive impact, please specify.
18. Do you have any other comments concerning the [ISF, BCF–CC Top-Up, NRT] initiative?
Return to Evaluation of Infrastructure Canada's Economic Action Plan Initiatives
- Date modified: