Evaluations - Evaluation of Infrastructure Canada's Economic Action Plan Initiatives - Annex 5

Annex 5 – Data Collection Instruments

Interviews

Note: The interview master list is presented to give the reader an overview of the main interview questions used for all EAP interviewees. For program/recipient-specific interview guides, please contact the department.

Question Indicator Group
Senior Managers Program Staff Delivery Partners Ultimate Recipients
Background
1.

[ISF, CC] To begin, please briefly describe your involvement with [ISF or BCFCC or Top-Up] and how long you have been involved. Are you/ were you involved in the delivery of other INFC programs? If so, which ones.

[NRT] To begin, please briefly describe your involvement with National Trails funding and how long you have been involved.

Initiative Objectives
2.

a) What in your view were the main objectives of the EAP initiative that you were involved in?

b) Were the main objectives clear to you?

c) In your view, are these objectives being met? If not what were the reasons for not meeting the objectives?

1.02

6.00 (performance question)

   
Project Approvals, Risk Management and Decision-making
Note: Questions in this section apply only to the process related to the initial March 31 deadline for this initiative and do not refer to the process related to project extensions under the EAP extension to October 2011.
3.

a) How were project eligibility criteria communicated?

b) Was the communication clear and timely?

c) What worked well and what needed improvement?

2.08
4.

Table 1 (provided separately) provides an overview of the project review and decision-making process for [ISF or BCFCC Top-Up or NRT]. In reference to Table 1:

a) This process was implemented with a view to balancing the need to expedite funding with the need to respect authorities and accountabilities. How effective was this process in achieving this balance?

Not Effective Somewhat Effective Effective Very Effective Don't Know
1 2 3 4  

Please provide examples to explain your response.

b) Was the approval process effective? What were the efficiencies or bottlenecks in processing project application (e.g. due diligence, capacity, etc.)?

c) Did the size, level, composition and operation of your main decision making bodies support effective decision-making? (SENIOR MANAGEMENT, STAFF)

d) What was your level of satisfaction with the quality of information available for decision-making and oversight (e.g. recommendations and supporting information)? (SENIOR MANAGEMENT)

Not Satisfied Somewhat Satisfied Satisfied Very Satisfied Don't Know
1 2 3 4  

Please provide examples to explain your response.

e) What was your level of satisfaction with the information available to assess and recommend projects (e.g. project application forms, certificates of construction-readiness)? (PROGRAM STAFF)

Not Satisfied Somewhat Satisfied Satisfied Very Satisfied Don't Know
1 2 3 4  

Please provide examples to explain your response.

2.03  
5.

[ISF, CC]

a) Was project assessment for approval done at the appropriate level within INFC? Please explain.

[ISF]

In some cases, an increase in spending authorities to the Minister meant that ISF projects were approved by the Minister and not by the Treasury Board. This was implemented in order to accelerate the approval process.

b) What effect did increased spending authorities have on the timeliness of project approvals for the process related to the initial March 31st deadline?

c) What effect did increased spending authorities have on the risk tolerance level at INFC in general?

d) Were decisions to approve ISF projects made at the appropriate level? Why or why not?

2.05

3.02
   
6.

Table 2 (provided separately) provides an overview of the tools and mechanisms employed to manage risk for [ISF or BCFCC Top-Up or NRT].

a) How were these tools and mechanisms used?

b) To what extent was risk information used as an input to decision-making?

c) Are there other tools, aside from the ones mentioned in Table 2, which were used to manage risk?

d) How useful and effective were these tools and mechanisms in managing risk given the requirement for accelerated approvals? What worked well and what needed improvement?

2.04

[not for ISF]

[CC only]

 
7.

Table 3 (provided separately) provides an overview of the requirements for [ISF or BCFCC Top-Up or NRT] project tracking.

a) How useful and effective were the data collection/reporting tools? (e.g. SIMSI, others)

b) Were there any challenges with respect to data collection and reporting requirements?

c) How was performance information used (to manage, to report)?

d) How useful and effective were these methods in project tracking taking into consideration the requirement for accelerated delivery? What worked well and what needed improvement?

2.04

[FDPs all]

[provinces a) and b) only]

 
8.

[ISF only]

a) Are you aware of any key reviews and audit observations for ISF?

b) Were you aware of any management action made in response to these observations?

c) Did the management actions respond adequately to the issues identified?

d) Were they implemented in a timely manner?

2.06

 

 
Partnerships and Stakeholder Engagement
9.

[ISF, CC]

a) Do you believe the roles and responsibilities between INFC and delivery partners were well defined?

b) Were there any overlaps between your roles and responsibilities and INFC or other partners?

c) What assisted/hindered you in understanding your role? (e.g. MOUs, contributions agreements, service level agreement, communication)

d) What worked well? What needed improvement?

[NRT]

a) Do you believe the roles and responsibilities between INFC and NTC organization were well defined?

b) Were there any overlaps between your role and that of INFC?

c) What assisted/hindered you in understanding your role? (e.g. MOUs, contributions agreements, communication)

d) What worked well? What needed improvement?

2.02

 
10.

a) Please describe any changes to the way INFC and delivery partners worked together, in reference to projects funded under the initial March 31st deadline, to enable expedited funding [ISF or BCFCC Top-Up or NRT]? How were these changes implemented?

b) Using the scale below, please indicate your level of satisfaction with how these changes were implemented.

Not Satisfied Somewhat Satisfied Satisfied Very Satisfied Don't Know
1 2 3 4  

Please explain your response. What worked well and what needed improvement?

2.10  
11.

Using the scale below, please indicate your level of satisfaction with the following:

Not Satisfied Somewhat Satisfied Satisfied Very Satisfied Don't Know
1 2 3 4  

(NB: Need to separately define "delivery partners" and "ultimate recipients" for each ISF/CC/NRT; not all questions will apply to all interview groups)

a) communication between INFC and delivery partners

b) communication between delivery partners and ultimate recipients

c) level of cooperation between INFC and delivery partners

d) level of cooperation between delivery partners and ultimate recipients

e) the division of roles and responsibilities between INFC and delivery partners

f) the division of roles and responsibilities between INFC and ultimate recipients (direct relationship) (Not NRT)

g) project approval processes for the initial March 31st deadline

i) project reporting requirements

j) funding mechanisms (i.e. memoranda of understanding, contribution agreements)

k) overall service and support provided by INFC

Please provide examples to support your opinions. What worked well and what needed to be improved?

2.10  
12.

a) Did your organization have appropriate capacity to respond to the accelerated delivery that the initiative demanded (e.g. number of staff, office space and equipment, skills/expertise) [ISF or BCFCC Top-Up or NRT]?

b) If not, what were the challenges and how were they overcome?

c) What challenges remain?

2.11

   

Management of Organizational Change
13.

a) What measures were taken by management to identify and implement organizational changes (e.g. new tools, HR, organizational structure, process change) at INFC to support rapid implementation of [ISF or BCFCC Top-Up or NRT]?

b) What worked well and what needed improvement? (senior managers and staff)

c) Were any additional changes implemented to accommodate the EAP extension to October 2011? How well did this work?

3.01

 

 
14.

a) In general, was INFC adequately staffed with skilled people in the right positions to support implementation of the initiative?

b) If not, how was necessary capacity built to overcome any barriers? (e.g. recruitment, training, tools, internal communications, etc.)?

c) What worked well and what needed improvement?

3.03


[a) and c) only]
 
15.

Using the scale below, please indicate your level of agreement with the following statements for the process related to the initial March 31st deadline.

Do Not Agree Somewhat Agree Agree Strongly Agree Don't Know
1 2 3 4  

a) I was part of the solution in delivering this rapid initiative.

b) I was engaged and informed about the organizational changes that were brought about to accommodate the requirement for accelerated delivery

c) I received clear direction in order to deliver on the initiative

d) My workload was manageable

e) The division of responsibilities was appropriate and effective

f) I was comfortable in my new role in delivering the initiative

g) I was comfortable making risk-based recommendations

h) I had the right skill set to perform my duties for [ISF or BCFCC Top-Up or NRT]

i) I received adequate training to perform my duties

Please provide examples to support your opinions. What worked well and what needed to be improved?

3.04      
16.

Using the scale below, please indicate your level of agreement with the following statements for the process related to the initial March 31st deadline.

Do Not Agree Somewhat Agree Agree Strongly Agree Don't Know
1 2 3 4  

a) Staff was part of the solution in delivering this rapid initiative.

b) Staff was engaged and informed about the organizational changes that were brought about to accommodate the requirement for accelerated delivery

c) Clear direction was provided to staff in order to deliver on the initiative

d) The workload of my staff was manageable

e) The division of responsibilities among staff was appropriate and effective

f) I was comfortable in my role in delivering the initiative

g) I was comfortable making risk-based decisions

h) Staff had the right skill set to perform their duties for [ISF or BCFCC Top-Up or NRT]

i) Staff was given adequate training to perform their duties

Please provide examples to support your opinions. What worked well and what needed to be improved?

3.04      
Effectiveness
17.

[ISF & CC only]

a) Did you apply for an extension to the original March 31, 2011 deadline?

b) What circumstances contributed to the need to extend the project?

c) Was the extension process effective (e.g. efficiencies or bottlenecks in the extension process)?

Please describe how or why it was or wasn't effective.

6.04

2.03

   

 

18.

Table 4 (provided separately) provides an overview of the delivery models for the initiative.

a) What advantages or disadvantages did each of these delivery models provide in relation to the requirement for rapid implementation? (e.g. were there bottlenecks, how could they have been unblocked, what factors facilitated or impeded delivery) Can you provide examples to support your opinion?

b) Are there features of these delivery models or other delivery models that should be included in future planning of similar initiatives as good practices?

8.01

 
19.

How could the initiative be improved or delivered differently by partners (federal, provinces, territories, and municipalities) to better satisfy the need for quick approval and delivery of funds?

8.02

20.

Are there any examples of challenges or unintended impacts as a result of the initiative?

7.01

21.

Is there anything else you wish to add?

Surveys

Note: The survey master list is presented to give the reader an overview of the main survey questions used for all EAP respondents. For program/recipient-specific survey guides, please contact the department.

Project Identification

1. Our records indicate that your organization has received funding for project(s) through the [ISF/BCFCC Top-Up/NRT] under Canada's Economic Action Plan. (drop down menu)

2. Please, identify up to three of the highest dollar value [ISF/BCFCC Top-Up/NRT]-funded projects which you are responsible for managing or administering. (Is there any common identifier that we can use to facilitate cross referencing)

 
 
 

3. When you think about the [ISF/BCFCC Top-Up/NRT]-funded projects for which you are responsible, please indicate:

  • How many projects are for new infrastructure? (pick list of numbers)
  • How many projects are for the rehabilitation or improvement of existing infrastructure? (pick list of numbers)

4. Which categories best describes the [ISF/BCFCC Top-Up/NRT]-funded project(s) for which you are responsible? (select all that apply)

[Categories will be separated based on each program's eligibility list]
















 

5. Did your organization also have unsuccessful applications for funding?



6. How would you categorize the type of organization for which you work: (select only one)







 

7. In which province/territory are/were your [ISF/BCFCC Top-Up/NRT]-funded projects located? (select one)










Include the following only for ISF/NRT



8. Please identify the type of agreement you entered into to obtain funding from the [ISF/BCFCC Top-Up]? Not for NRT



 

Stakeholder Relationships

9. Please complete the questions below regarding stakeholder relationships by indicating your level of agreement with the following statements:

  Strongly Disagree Disagree Agree Strongly Agree Not Applicable Do Not Know
9.01(2.10e) The roles and responsibilities of INFC were clear to me [ISF only].
9.02 (2.10e) The roles and responsibilities of the provinces/territories were clear to me [ISF, BCFCC only].
9.02 (2.10e) The roles and responsibilities of the NTC were clear to me [NRT only].
9.02(2.10a) I am satisfied with the level of advice and support I received from the province or territory with which I had an agreement. [ISF, BCFCC only]
9.02(2.10a) I am satisfied with the level of advice and support I received from NTC with which I had an agreement. [NRT only].
9.03 (2.10a) I am satisfied with the level of advice and support I received from INFC (ISF direct agreements only) [ISF]
9.05 (2.10) How could stakeholder relationships be improved?

Project Application and Approval

10. Please complete the questions below by indicating your level of agreement with various aspects of project application, approval and reporting [not Quebec]:

  Strongly Disagree Disagree Agree Strongly Agree Not Applicable Do Not Know
10.01 (2.08) Availability of information on project eligibility was timely.
10.02 (2.08) Information on project eligibility was clear.
10.03 (2.08) I understood which projects would be eligible for funding.
10.04 (2.10b) Project approval decisions were communicated in a timely manner.
10.05 (2.10b) Project approval decisions were clear.
10.06 (2.10b) I understood why my projects were accepted.
10.07 (2.10b) I understood why my projects were rejected.
10.08 (2.10b) The project reporting requirements were clear.
10.09 (2.10b) I understood my role and responsibilities related to reporting.
10.10 (2.10b) Reporting requirements were not overly burdensome.
10.11 (2.10b) Overall, I am satisfied with the project approval process.
10.12 (2.10b) Overall, I am satisfied with the project reporting requirements.
10.13 How could the project application and approval process be improved?

Project Implementation

11. Are all you projects completed?


12. (2.09, 2.10, 2.11) Did you experience delays or do you anticipate delays in the implementation of your project(s) such that project(s) can not be completed by the original March 31, 2011 deadline?


 (skip to Q15)

13. When do you expect your project(s) to be completed?




14. What are the main reasons that the project(s) were/will be delayed? (select all that apply)





 (additional question at Q16)




 

15. (2.11) To what extent did the [ISF/BCFCC Top-Up/NRT] initiative, with its short time frames, affect you?

Significant
Negative Impact
Negative Impact Positive Impact Significant
Positive Impact
Don't Know Not Applicable

Go to Q16

Go to Q16 Go to Q17 Go to Q17 Go to Q18 Go to Q18

16. (2.11) If you identified any negative impact, please identify the issues. (select all that apply)





 

17. If you identified a positive impact, please specify.

18. Do you have any other comments concerning the [ISF, BCFCC Top-Up, NRT] initiative?

Return to Evaluation of Infrastructure Canada's Economic Action Plan Initiatives

Date modified: