Audit of Information Technology Contracts

November 2011

Table of Contents

  1. Executive Summary
  2. Background
  3. Audit Objective
  4. Audit Scope
  5. Audit Approach
  6. Audit Findings
  7. Audit Opinion
  8. Statement of Assurance

Appendix A: List of Audit Criteria

1 Executive Summary

Infrastructure Canada's (INFC) Information Management/Information Technology (IM/IT) Division has entered into two contracts to provide service management support for the supply, installation, operation, and development of the Shared Information Management System for Infrastructure (SIMSI) and to provide professional services to meet its operational needs. Federal funds of $43.6 million and $12.9 million, respectively, have been committed for these contracts. IM/IT manages the monitoring and invoice verification associated with each contract.

The overall objective of the assurance engagement was to provide a reasonable level of assurance that contract management practices were in place to support the relationship management activities with the contract vendors, in that these were appropriate and effectively supported the achievement of INFC's Information Technology (IT) strategy, ensured compliance with IT contracting policies and processes (as defined by the Government of Canada and by IT industry standard best practices), and mitigated the areas of highest risk.

The audit was conducted in accordance with the Internal Auditing Standards for the Government of Canada and the International Standards for the Professional Practice of Internal Auditing. Based on the audit evidence gathered to support the findings, overall, good contract management practices were found to be in place.

The audit has also noted the following three areas where the Management would benefit from strengthening its management practices:

  • Documentation standards require strengthening to support the work that is already performed.
  • There is a need for additional oversight procedures in the professional services invoice verification process.
  • There is an opportunity to build internal capacity within IM/IT while reducing the overall operating cost to INFC.

Management is in agreement with the audit findings and recommendations. Management has developed action plans to address the recommendations and these action plans have been included in the report.

Original signed by

Inanc Yazar
Director of Internal Audit for Infrastructure Canada

Date

Laura Ruzzier
Chief Audit and Evaluation Executive
for Infrastructure Canada and Transport Canada

Date

2 Background

The Shared Information Management System for Infrastructure was built shortly after the announcement of the Infrastructure Canada Program (ICP) in 2001 to support the implementing agencies and INFC in their responsibilities to administer the ICP and subsequent programs. SIMSI provides the information technology support system to assist program managers and stakeholders (municipalities, provinces and federal implementing agencies) in managing thousands of projects throughout their project life cycle.

A contract was awarded in October 2006 for the supply, installation, operation, and development of SIMSI. The service management support contract covered a period of 3 years. Following the exercise of two one-year extensions, INFC sought and received a one-year extension to October 2012 from Public Works and Government Services Canada and Treasury Board in an effort to continue its commitment to meet its program objectives following the Government of Canada's 2009 Economic Action Plan (EAP). Federal funds of $43.6 million have been committed under the contract.

As funding for INFC programs increased, supporting the needs of INFC necessitated a rapid increase in required IM/IT support services and products. As a result of this pressure, the Chief Information Officer identified a requirement for expert information technology consultants to enhance the capabilities of IM/IT (both technically and managerially). In July 2007 a contract for professional services was entered into with a second vendor to assist in the oversight of design, development, enhancement, maintenance, and operational activities, all on an as and when required basis. These services were intended to help INFC to support the SIMSI application, to help with the development and support of non-SIMSI requirements such as intranet/internet development work and collaboration, and to help with the support of on-going operations such as change management and development of service level standards.

The initial contract period with the vendor of professional services began in July 2007 when the contract was issued for a period of 3 years. It has been extended twice to June 2012. Federal funds of $12.9 million have been committed under the contract.

3 Audit Objective

The overall objective of this assurance engagement was to provide a reasonable level of assurance that contract management practices are in place to support the relationship management activities with the contract vendors, in that these are appropriate and effectively support the achievement of INFC's IT strategy, ensure compliance with IT contracting policies and processes (as defined by the Government of Canada and by IT industry standard best practices), and mitigate the areas of highest risk.

Within this overall objective, the following sub-objectives were identified and aligned to the risks for the management of IT contracts, identified as part of our risk assessment:

Sub-Objective 2.1: In-scope services are provided as described in the contract within the definitions of quality, quantity, and functionality (application and operation processes perform/operate as intended).

Sub-Objective 2.2: Services are billed accurately by the vendor according to the method of payment specified in the contract; adjustments to the payment are based upon performance issues and are communicated to the vendor on a timely basis; payments are made to the vendor on a timely basis; and adjustments to in-scope activities (change requests and/or contract amendments) are documented and approved in advance of work being undertaken.

A list of Audit Criteria is available in Appendix A.

4 Audit Scope

The scope of the audit involved an examination of the contract management practices in place to support the vendor relationship management activities of the service management support and professional services contracts for the period of April 1, 2008 to March 31, 2011. More specifically, this included an examination of the practices in place with respect to approval of task authorizations (TA) by IM/IT to request services under the contracts as well as contract items included under fixed-price arrangements, monitoring, reporting, and payment.

The audit did not include an evaluation of the appropriateness of the technical requirements of the contract or its task authorization deliverables.

5 Audit Approach

The audit was conducted in accordance with the Internal Auditing Standards for the Government of Canada and the International Standards for the Professional Practice of Internal Auditing.

The examination phase of the audit began in July 2011 and was substantially completed in August 2011. The examination employed various techniques including interviews, review of financial and non-financial documentation, and analytic review. The audit criteria were based on the Treasury Board Contracting Policy, The Information Systems Audit and Control Association's (ISACA) Outsourced IT Environments Audit/Assurance Program, and ISACA's Control Objectives for Information Related Technology (CobiT) framework.

A risk-based judgmental sample methodology was used to determine the sample size and select the task authorizations and invoices to be examined during the audit. The sample targeted TAs and invoices based on:

  • the materiality of the TA, its amendments, and individual invoices;
  • the number of amendments to each TA and the effects of those changes;
  • the complexity of the deliverables; and
  • the terms and conditions of payment.

The sample served as an indicator of the quality of the management practices in place to support the vendor relationship management activities. Since this is not a statistical sample, the results cannot be extrapolated to all invoices within IM/IT.

For purposes of this report, the residual risk rankings associated with findings used a low, moderate, high three-point scale and were subjectively judged based on our knowledge of the IT contracts gathered during the audit. The subjective criteria are:

High Threats/Opportunities have very significant impact on INFC's objectives, are imminently likely and no, or uncertain mitigation measures are in place.
Moderate Threats/Opportunities have significant impact on INFC's objectives, have a longer-term likelihood and reliable mitigation measures are planned or are being established.
Low Threats/Opportunities do not have a significant residual risk to INFC's objectives.

6 Audit Findings

6.1 Monitoring

The Treasury Board Policy on Internal Control states that Parliament and Canadians expect the federal government to be well managed with the prudent stewardship of public funds, the safeguarding of public assets, and the effective, efficient, and economical use of public resources.

It was expected that IM/IT would monitor the vendor performance and the performance of each TA while Finance and Administration Division would monitor the authorities required for each commitment and payment of federal funds.

Audit Observation - Monitoring

Observation
Low Risk
Performance Monitoring

Service Standards - INFC and the vendor of the service management support contract agreed to acceptable service standards in the Service Level Agreement (SLA). The agreement defines specific performance criteria that will be evaluated and an acceptable performance threshold for each criterion. It was expected that IM/IT would monitor the vendor performance using the service standards defined in the SLA on a periodic basis.

It was found that IM/IT monitored the vendor's performance against most of the service standards on a fairly consistent basis. While the vendor reports used by IM/IT for monitoring purposes did not include a few of the service standard criteria that were agreed to in the SLA, the monitoring was still fairly comprehensive.

Executive Scorecard - IM/IT developed Executive Scorecards to provide the vendor of the service management support contract with feedback on their overall performance. It was found that the scorecards were mostly produced on a regular basis and that the format of the scorecard was fairly consistent between assessments.

Monitoring during Invoice Verification

Invoice Verification procedures – IM/IT implemented an invoice verification process to support their recommendation of the invoice for payment. The process includes an assessment of financial and technical factors. It was found that all work associated with the invoices was performed under approved and active TA. Overall the documentation supported that the progress was commensurate with the work invoiced at that time. For each TA, the named resources were all approved resources. The rates used to calculate the amount of the invoice were in accordance with the approved cost estimates and contracts. Finally, completed TAs were consistently closed with an amendment and unused federal funds were decommitted.

Financial Monitoring

Invoice Challenge Function – IM/IT employed a challenge process for vendor invoices that triggered a potential financial or technical concern. It was found that for invoices where IM/IT did identify possible concerns, there was a process in place to address the concerns with the vendor and to resolve them prior to invoice payment. The documentation to support the process was sometimes not available.

Financial reconciliation - IM/IT implemented a process to reconcile its invoice management database to the department's financial records on a monthly basis. It was observed that the approval and reconciliation were performed and documented on a monthly basis.

Financial transaction approvals – INFC complied with the Financial Administration Act (FAA) using a documented Section 32 approval for the commitment of federal funds, a Section 33 approval with respect to the payment, and a Section 34 approval that work was performed in accordance with the contract. It was found that the commitment of funds associated with the contracts was appropriately approved with FAA Section 32 commitment. While, overall, the invoices were appropriately approved with FAA Section 33 and Section 34 approvals, it was found that there was insufficient delegated authority for some FAA Section 33 and Section 34 approvals. It was later observed that these approvals were re-performed appropriately prior to the end of the conduct phase of the audit.

The financial transactions were able to be approved inappropriately due to the limited oversight procedures in the existing approval verification process.

The impact of invoice payments being made with inappropriate FAA Section 33 or 34 approvals may cause an increase in the risk of federal funds being used inappropriately.

Recommendations Management Action Plan
1.1 It is recommended that Finance and Administration Division supplement their existing approval verification process with increased oversight. 1.1 Agreed. Oversight has been improved through enhanced procedures in Finance and Administration to verify delegated approvals prior to processing requests for payment. A printed copy of the verification checklist has been given to each individual in Accounting Operations who processes payments to post at their workstations, to reinforce the principles of verification, including a review of delegations of authority for section 33. Finance and Administration sign-off on requests for payment will indicate that this verification has been completed. For all requests for payment greater than $100,000, it is mandatory that a hard copy of the checklist be printed and retained with all other related documentation. On a quarterly basis, the Head, Accounting Operations & Financial Systems will complete a formal review of delegated authorities for all transactions over $100,000, to confirm financial authorities were properly exercised, and report findings to the Manager, Accounting Operations & Financial Systems.
Assistant Deputy Minister Responsible: Su Dazé
Due Date: March 31, 2012

6.2 Documentation

The Treasury Board Policy on Information Management requires that the Deputy Heads are responsible for ensuring that decisions and decision-making processes are documented to account for and support the continuity of departmental operation, permit the reconstruction of the evolution of policies and programs, and allow for independent evaluation, audit, and review. In order to support the Deputy Minister in her accountabilities, it was expected that the TA and invoice verification documentation would consistently include all key documents.

Audit Observation - Documentation

Observations
Low Risk
Invoice Verification

The contract for service management support and the contract for professional services each used TAs to define the work that is to be performed, the resources required, and the associated cost. Once monthly invoices for all work performed were submitted by the vendors, IM/IT performed an invoice verification to monitor the progress of each TA and to ensure that the invoice could be recommended for approval under Section 34 of the Financial Administration Act.

It was found, during the audit, that not all key documents to support the invoice verification process could be located by IM/IT for all invoices. The attestation of the invoice verification performed by the project officer could not be located for some invoices. Specific cost estimates to support that the invoice used approved per diem rates could not be located. Documentation to support that the invoiced amount of work was commensurate with progress could not be consistently provided to support the attestation of invoice verification. Documentation to support that the deliverable associated with the work was completed, as per the invoice, could not be provided. On some TAs, the deliverable was not clearly stated.

For the fixed price TAs, IM/IT implemented a process of invoice verification that was tailored to the nature of the fixed price component of the contract. It was found that the evidence of completion of the invoice verification process for these invoices was consistently documented.

When IM/IT identified a possible technical or financial concern with an invoice, the challenge process was used to communicate and resolve the issue with the vendor prior to payment. It was found that the documentation to support the process was not consistently available.

Vendor Performance

INFC and the vendor of the service management support contract agreed to various service standards in a SLA. The SLA identified several performance criteria and acceptable performance standards that were used to evaluate the vendor's performance. Given the technical nature of the performance criteria and the volume of transactions that must be monitored, the vendor programmed alerts into its software to track the statistics and to report on any occurrences when performance fell below the acceptable levels. It was found that IM/IT monitored the status of significant occurrences and ensured that they could be resolved without further escalation.

The reason that documentation sometimes could not be located or was inconsistent, between similar invoices, was that there were no written policies or procedures to ensure that work was performed consistently by different project officers or by the same project officer at different times.

Incomplete and inconsistent documentation may not allow the Deputy Minister to defend the amount of federal funds that have been paid to the vendor.

Recommendations Management Action Plan

2.1 It is recommended that IM/IT formalize its existing invoice verification and approval process by preparing written procedures. IM/IT may consider including file checklists in the procedures to ensure that all key documents are kept on file.

2.1 Agreed. IM/IT will formalize its invoice verification procedures to reflect the procedures currently in place within IM/IT. In addition, process maps that were prepared by Internal Audit in consultation with IM/IT during the course of this audit will be incorporated with file checklists into the written documentation.
Assistant Deputy Minister Responsible: Su Dazé
Due Date: December 31, 2011

6.3 Oversight

The Treasury Board Policy on Internal Control states that Parliament and Canadians expect the federal government to be well managed with the prudent stewardship of public funds, the safeguarding of public assets, and the effective, efficient, and economical use of public resources. The deputy head is responsible for ensuring the establishment, maintenance, monitoring and review of the departmental system of internal control to mitigate risks in the following broad categories:

  • The effectiveness and efficiency of programs, operations and resource management, including safeguarding of assets;
  • The reliability of financial reporting; and
  • Compliance with legislation, regulations, policies and delegated authorities

It was expected that IM/IT would implement adequate oversight procedures to ensure that more than one person would be involved in the approval of key supporting documents, invoice verification, recommendation for payment, and FAA Section 34 approval.

Audit Observation - Oversight

Observation
Low Risk
Vendor invoices for professional services

IM/IT used the professional services contract to meet its organizational needs. Consultants were paid by the contract vendor. They worked alongside INFC employees and their timesheets were approved by their immediate INFC manager.

It was found that invoices from the vendor that provided professional services were not subject to adequate oversight. The process for those specific invoices allowed for the same person to sign the supporting timesheets, perform the invoice verification, and recommend the invoice for payment. In one instance, it was found that the same person who performed the invoice verification and made the recommendation for payment also provided the FAA Section 34 approval.

Invoices from both vendors

For the invoices from the vendor who provided support management services and the vendor who provided professional services, it was found that the recommendation for payment was not always provided to the person who signed the FAA Section 34 approval.

The reduced oversight of professional services invoices resulted from the additional restrictions to the invoice verification process that were required to ensure that the per diem rates paid by INFC to the vendor remained confidential. The process was modified to limit involvement in the process to the consultant's immediate manager and the manager of the fund centre whose funds were used to pay the invoice. When the consultant's manager and the fund centre manager were the same person, oversight was further reduced. With respect to invoices from both vendors, recommendation was not provided prior to approval of the invoice due to the lack of documented procedures for the invoice verification and approval process.

The potential impact of reduced oversight during IM/IT's invoice payment process is that financial risks associated with invoice payment process may not be mitigated. The lack of segregation of duties within the professional service invoice payment process may not mitigate basic financial risks involved in approving payment of funds. Furthermore, for invoices from both vendors, it may be difficult to defend the FAA Section 34 approval without a documented recommendation for payment.

Recommendations Management Action Plan
3.1 See Recommendation 2.1 in Section 6.2 Documentation. Not Applicable
Manager Responsible:  
Due Date:  
Recommendations Management Action Plan

3.2 For the invoices related to the professional services contract, the process should still allow IM/IT to restrict access to the per diem rates paid to the vendor. It is recommended that invoice verification and recommendation be performed by two different managers who already have access to the per diem rates.

It is also recommended that the invoice verification approval process require different people to recommend and approve the invoice.

3.2 Agreed. Access to per diem rates is given only to those with a need to know. We will ensure that invoice verification, recommendation, and approval be done by at least two different persons. This will be added to the written documentation related to these processes.
Assistant Deputy Minister Responsible: Su Dazé
Due Date: December 31, 2011

6.4 Role of Professional Service Contract

The Clerk of the Privy Council articulated the following approaches as part of his short-term to medium-term approaches for all departments in the 2010-2011 Public Service Renewal Action Plan:

  • Recruitment: The Public Service needs to hire talented graduate and mid-career recruits who reflect Canada's diversity and fill skills gaps that have been identified through integrated planning.
  • Development: The foundation to delivering on our business is to improve our approaches to learning and managing talent, succession planning and performance with an emphasis on middle management and knowledge transfer to ensure transition of the 'management ethos' to succeeding generations.

It was expected that IM/IT would develop key competencies within its own staff to provide the required internal capacity to meet its objectives. Given the value, technical complexity, and strategic importance of the SIMSI application, it was expected that IM/IT would build the internal capacity required to assess the technical nature of the contract deliverables. It was also expected that the capacity for strategic planning would be developed internally.

Audit Observation - Role of Professional Service Contract

Observation
Low Risk

IM/IT has used the professional services contract to fill its organizational needs as required.

It was found that the professional services consultants filled a variety of IM/IT roles that ranged from project management support to portal architecture to Senior Management support. While some consultants filled SIMSI related positions, other consultants provided professional services for strategic planning and for development of internal IM/IT projects.

It was found that while the TAs that defined the work required under the contract were initially for short periods of time, the TAs were amended repeatedly to extend the end date of the work. One TA with multiple amendments eventually covered three years of work.

The reliance of IM/IT on consultants to fulfill its organizational needs was due to difficulties in forecasting the resources required to meet the evolving operational needs of INFC's rapid growth and the increased demands of the Economic Action Plan transfer payment programs.

The impact of using contractors to fill long-term positions within IM/IT is that significant financial resources have been and continue to be spent on vendor contracts. In addition, the corporate knowledge developed by consultants may not be retained by IM/IT.

It is recognized that IM/IT has taken steps to reduce their reliance on the professional service contract by using the Interchange Canada process to hire private sector IM/IT professionals for up to 4 years at a time.

Recommendations Management Response

4.1 It is recommended that IM/IT focus on building internal capacity using indeterminate employees when possible and term employees when the need is for a limited period of time. INFC will benefit from significant operational savings overall if IM/IT reduces its contract budget and builds up its internal capacity to meet its organizational needs.

4.1 Agreed. Prior to exercising existing or new procurement vehicles for professional services, management will assess its short-term and long-term needs, and will complete a financial analysis to justify the use of contracted resources in place of indeterminate or term employees.

Management will also assess its long term resource needs and will develop a strategy to build internal capacity in order to retain corporate knowledge within IM/IT and/or to create additional indeterminate positions to address ongoing business requirements. The strategy will include internal capacity development goals with deliverable dates.

Assistant Deputy Minister Responsible: Su Dazé
Due Date: March 31, 2012

7 Audit Opinion

Based on the audit evidence gathered, it was found that, overall, good contract management practices are in place to support the relationship management activities with the contract vendors, in that these are appropriate and effectively support the achievement of INFC's IT strategy, ensure compliance with IT contracting policies and processes (as defined by the Government of Canada and by IT industry standard best practices), and mitigate the areas of highest risk.

The audit has also noted the following three areas where the Management would benefit from strengthening its practices:

  • Documentation standards require strengthening to support the work that is already performed.
  • There is a need for additional oversight procedures in the professional services invoice verification process.
  • There is an opportunity to build internal capacity within IM/IT while reducing the overall operating cost to INFC.

8 Statement of Assurance

In the professional judgment of the Chief Audit and Evaluation Executive, sufficient and appropriate audit procedures have been performed and evidence gathered to support the accuracy of the conclusions reached and contained in this report. The audit findings and conclusions are based on observations and analyses of conditions as they existed on the audit date, against established criteria agreed upon with management.

The findings are only applicable to the entity examined. The evidence was gathered in accordance with the Internal Auditing Standards for the Government of Canada and the International Standards for the Professional Practice of Internal Auditing.

Appendix A: List of Audit Criteria

Audit Criteria

1.1.1 Performance Delivery:

Delivery performance is monitored by INFC using Service Level Agreements, Statements of Work and/or other performance indicators on a routine and frequent basis; performance issues are reported to the vendor on a regular basis.

1.1.2 Delivery Review:

INFC reviews overall vendor performance on a regular basis and submits status performance reports to INFC senior management.

1.1.3 Application Processes:

Application processes are monitored by INFC IM/IT and evaluated for functionality (application processes perform as intended).

1.1.4 Operation Services:

Operation processes are monitored by INFC IM/IT and evaluated.

1.2.1 Billing for Products and Services:

Billing is reviewed prior to payment for accuracy and comparison to budget, and reflects adjustments initiated by the challenge process, and approval in accordance with FAA Sections 32, 33, and 34.

Date modified: