Audit of the PIA Review Process

Prepared by Deloitte & Touche LLP and affiliated entities
for the Office of the Privacy Commissioner of Canada

July 20, 2012


Executive Summary

Background and Context

The Office of the Privacy Commissioner of Canada (OPC) is an Agent of Parliament mandated to oversee compliance with both the Privacy Act, which covers the personal information-handling practices of federal government departments and agencies (hereafter referred to as institutions), and the Personal Information Protection and Electronic Documents Act (PIPEDA), Canada's federal private-sector privacy law.

As outlined in the OPC’s 2011-2012 Report on Plans and Priorities (RPP), OPC’s strategic outcome is to ensure that the privacy rights of individuals are protected. Part of the OPC’s activities includes reviewing Privacy Impact Assessments (PIA) that are conducted by federal government institutions that are subject to the Privacy Act and the Treasury Board (TB) Directive on Privacy Impact Assessment (PIA).  This Directive requires government institutions to ensure that privacy implications are appropriately identified, assessed and resolved before a new or substantially modified program or activity involving personal information is implemented.  The OPC does not approve or endorse PIA submissions, but rather reviews them and makes recommendations to the institutions for improvement. 

A PIA review group made up of six full-time equivalents (FTEs) within the Audit and Review Branch has the lead responsibility within the OPC for reviewing PIAs and providing recommendations to assist organizations in ensuring that they have addressed risks to privacy. For fiscal year 2011-12Footnote 1 , 49 PIAs were received by the OPC, and 54 PIAs were reviewed and a recommendation letter sent to the submitting institutions.  There is a backlog of 43 PIAs that remain to be reviewed. 

The purpose of this audit engagement is to provide assurance on the effectiveness of risk management, controls, and governance processes that support the PIA review process, with a specific focus on the quality assurance process supporting information technology (IT)-oriented PIAs that are complex in nature.

Summary of Findings

The key findings with regards to the audit are provided below.

Strengths

  • The PIA review process provides federal government institutions with understandable and actionable recommendations and advice, linked to the Privacy Act and leading privacy practices.
  • For high priority IT-oriented PIAs that are complex in nature, the PIA review group ensures that the appropriate stakeholders within the OPC are consulted for advice during the PIA review process.
  • Processes have been developed to ensure the focus of reviews is on PIAs related to those initiatives that have the most impact on the privacy of Canadians through a triage process that assesses a PIA against the OPC’s four strategic policy priorities, as well as other relevant factors.
  • The PIA review group regularly informs other OPC branches on privacy issues related to PIA submissions through participation on several committees and through a bi-weekly ‘PIA Files of Interest’ document that is provided to OPC senior management through the Privacy Working Group (PWG).
  • The PIA review group have increased its consultation and outreach within the federal government privacy community, becoming involved earlier in high priority PIAs, including those that are complex in nature; publishing the ‘PIA Expectations Guide’; and, offering workshops on PIAs which include the Treasury Board Secretariat (TBS) as a participant.
  • The PIA review group provides training to staff and management of the branch in several forms, allowing the PIA review group to build the required competencies and skills.

Findings

  • PIA review files do not consistently document all the analysis/input that was used to develop recommendations.
  • Current performance measurement should be improved.
  • Roles and responsibilities related to the PIA review process should be further defined throughout the OPC.

Conclusion

Based on the aforementioned observations and overall scope of the audit, the OPC has moderate issues related to the effectiveness of its current risk management, controls, and governance processes that support the PIA review process, including the quality assurance process supporting IT-oriented PIAs that are complex in nature.  The recommendations included in this report are intended to strengthen the risk management, controls, and governance processes that support the PIA review process.  Management responses are included at the end of each finding.

This report and audit were conducted for OPC management purposes.  Use of this report for other purposes may not be appropriate.

Statement of Assurance

In our professional judgment, sufficient and appropriate audit procedures have been conducted and evidence gathered to support the accuracy of the opinion provided and contained in this report. The opinion is based on a comparison of the conditions, as they existed at the time, against pre-established audit criteria that were agreed with management. The opinion is applicable only to the processes examined. The evidence was gathered in compliance with Treasury Board Policy, Directives, and Standards on internal audit for the Government of Canada. The evidence has been gathered to provide senior management with reasonable assurance of the accuracy of the conclusions drawn from this audit.

Audit Objective, Scope and Approach

Background

The OPC is an Agent of Parliament mandated to oversee compliance with both the Privacy Act, which covers the personal information-handling practices of federal government institutions, and PIPEDA, Canada's federal private-sector privacy law.

The OPC’s strategic outcome is to ensure that the privacy rights of individuals are protected. As an advocate for the privacy rights of Canadians, the Privacy Commissioner’s powers include:

  • Investigating complaints, conducting audits and pursuing court action under the two federal laws identified above;
  • Publicly reporting on the personal information-handling practices of public- and private-sector organizations;
  • Supporting, undertaking and publishing research into privacy issues; and,
  • Promoting public awareness and understanding of privacy issues.

As well, the OPC has four strategic priorities which have been identified for 2011-2012.  These priorities are updated annually.  The four strategic priorities are:

  • Identify, adopt, and deliver on new service delivery models to maximize results for Canadians;
  • Provide leadership to advance the four priority privacy issues (information technology, public safety, identity integrity and protection, and genetic information);
  • Support Canadians, organizations and institutions to make informed privacy decisions, both nationally and internationally; and,
  • Enhance and sustain organizational capacity.

The OPC also identified four strategic policy priorities to focus its approach to emerging privacy issues over the coming few years. These four strategic policy priorities are:

  • Information technology;
  • National security;
  • Identity integrity and protection / identity theft; and,
  • Genetic information.

The TB Directive on Privacy Impact Assessment (PIA) require government institutions to ensure that privacy implications are appropriately identified, assessed and resolved before a new or substantially modified program or activity involving personal information is implemented.  TBS has produced tools and guidance to support the preparation of PIAs, which are to be prepared at the earliest possible stage in the development of affected initiatives. The objective of a PIA is to identify potential privacy risks and to devise strategies to eliminate or mitigate these risks.  Federal government institutions are required to provide a copy of the approved PIA to TBS, with a copy being provided at the same time to the OPCOPC does not approve or endorse PIA submissions, but rather reviews the PIAs in order to make recommendations to institutions to improve their privacy practices.  The Audit and Review Branch has the lead responsibility within the OPC for reviewing PIAs.   The Branch is managed by the Director General (DG), Audit and Review, and includes 20 FTEs, with an annual budget of approximately $1.9 million ($1.5 million for salaries, $400k of other expenses).  The PIA section of the Branch is co-managed by two PIA Review Managers who are supported by four PIA Review Officers for a total complement of six FTEs.

Refer to Appendix B for an overview of the PIA review process, including the quality assurance processes that support IT-oriented PIAs that are complex in nature.

Audit Objective

The purpose of this audit engagement is to provide assurance on the effectiveness of risk management, controls, and governance processes that support the PIA review process, with a specific focus on the quality assurance process supporting IT-oriented PIAs that are complex in nature.

Audit Scope

The audit included assessing the following aspects of the PIA Review process:

  • Organization and management structure;
  • Plans established to support the achievement of objectives;
  • Performance management program supporting the activity;
  • Professional development and reliance on other subject matter experts; and,
  • The quality assurance and improvement process in place. 

This audit was conducted with particular focus on the quality assurance process supporting complex PIAs, where advanced technical advice is received, in particular as it relates to IT-oriented programs or activities; an analysis of whether OPC is ensuring that the advice provided is adequately assessed by OPC management, appropriately interpreted within the context of the Privacy Act and supports sound recommendations.

Audit Approach

The approach and methodology used for this audit was consistent with the Internal Audit standards as outlined by the Institute of Internal Auditors (IIA), and is aligned with the Internal Audit Policy for the Government of Canada (GC).

As an Agent of Parliament, OPC works independently from the Government of Canada and is therefore not obligated to follow the management improvement initiatives put forward in the Federal Public Service.  Nevertheless, the OPC strives to maintain a control framework that is reflective of industry leading practices.  Consequently, the framework of Core Management Controls and Audit Criteria (CMC) established by the Office of the Comptroller General of Canada (OCG), and the Management Accountability Framework (MAF VII) were leveraged to develop the audit criteria detailed in Appendix C.  Other criteria were also included to ensure appropriate coverage of the aforementioned audit scope.

Based on risks identified throughout the planning phase of the audit, a risk-based audit program was developed to detail on how the audit objective, criteria and risks were addressed.  The audit program includes the following audit procedures:

  • Review of OPC Report on Plans and Priorities and the Audit and Review Branch Business Plan.
  • Review of the OPC Monthly Scorecard Report.
  • Review of relevant committee and working group documents.
  • Review of tools, templates and numerous other documents related to the PIA Review process.
  • Review of a sample of PIA Review files in which a recommendation letter was sent from the OPC to federal government institutions since April 2010.  A total of 12 files were selected, seven were high priority files and five were low priority files.  Furthermore, of the 12 files selected, seven (a mixture of both high and low priority) were from April 2011 onwards.
  • Interviews with individuals from across the OPC (refer to Appendix A).

The audit was conducted within the following timelines:

  • Planning Phase : December 2011
  • Examination Phase: January 2012 - February 2012
  • Reporting Phase: March 2012
  • Presentation to the OPC Audit Committee: May 2012

Findings and Recommendations

Strengths Noted

The following strengths were noted with regards to the PIA review process:

  • The PIA review process provides federal government institutions with understandable and actionable recommendations and advice, linked to the Privacy Act and leading privacy practices. 
  • For high priority IT-oriented PIAs that are complex in nature, the PIA review group ensures that the appropriate stakeholders within the OPC are consulted for advice during the PIA review process, including technology analysts from the Technology Analysis Branch (TAB).
  • Processes have been developed to ensure the focus of reviews is on PIAs related to those initiatives that have the most impact on the privacy of Canadians.  This is accomplished through a triage process that assesses a PIA against the OPC’s four strategic policy priorities, as well as other factors (e.g., the sensitivity of the privacy issue, number of people affected).   Based on initial triage, PIAs considered high priority are reviewed in greater detail, including IT-oriented PIAs that are complex in nature.   
  • The PIA review group regularly informs other OPC branches on privacy issues related to PIA submissions. The PIA review group is represented at the Privacy Working Group (PWG), which meets every two weeks, and on each of the four committees related to the OPC’s four policy priorities.  Monthly meetings are held with staff from other units, including the policy group (within Legal, Policy and Research Branch), Privacy Act investigators from the Inquiries and Investigation Branch, and  technology analysts from TAB. A bi-weekly ‘PIA Files of Interest’ document is also circulated to the PWG.
  • The PIA review group have increased its consultation and outreach within the federal government privacy community.  Proactively, the PIA review group are attempting to be involved earlier in high priority complex PIAs to help submitting institutions address privacy issues early during the development and design of initiatives; this also helps ensure better quality PIAs are submitted to the OPC.  Furthermore, a “PIA Expectations Guide” has been published for federal government institutions, and joint workshops between the OPC and TBS have been held for the federal government privacy community. 
  • Considering the breath of PIA subject areas, the PIA review group has recognized the importance of building and maintaining staff capabilities and knowledge, and provides training to staff and management of the branch in several forms. Orientation training is provided to new staff and ongoing training is provided in the form of  internal training workshops on topics such as project management and privacy, as well as having staff attend external privacy conferences / seminars which is reviewed and approved as part of staff’s annual Training Plan.

Audit Findings

Finding 1: PIA review files do not consistently document all the analysis/input that was used to develop recommendations

PIA review files do not consistently contain the necessary documents/information in order to provide an understanding on the actions that were taken or the input received, which lead to the recommendations provided by the OPC in response to PIA submissions by an institution. For example, through discussion with staff in other Branches, it was evident that the PIA review group extensively consults with their colleagues in other branches on PIAs that are complex in nature; however, in only one of the PIA files reviewed by the audit team was there evidence on the file of the input provided to the PIA Review Group by another branch.

Several templates have been developed to assist staff in the performance of PIA reviews (e.g., acknowledgement letter, triage sheet, PIA review template, recommendation or low priority letters). An analysis of PIA review files indicated that completed templates are not always retained on a PIA review file (for example, two files out of the total of 12 files reviewed, did not have an acknowledgement letter or triage sheet). The recommendation letter template, which acts as a main formal communication channel between the OPC and federal government institutions is not used in a consistent fashion. An analysis of recommendation letters indicated that some letters clearly indicated recommendations and requests for clarification; while others had these points dispersed throughout the narrative of the letter, making it more difficult to quickly grasp the actions requested by the OPC.

In the case of the PIA review template, which was developed to assist PIA Review Officers in reviewing PIAs and analyzing the issues related to them, the template is not always completed (of the seven high priority files reviewed, the template has been completed for four of them). Without the template being complete, there is often no analysis on a file, or a framework to allow a reviewer to understand the issues considered by a PIA Review Officer, and the reasoning for what ultimately was included/not included in a recommendation letter.

Impact

Not having sufficient documentation in a file to understand the analysis/input that was used to reach the conclusions outlined in the recommendation letters to federal government institutions may make the quality assurance and review process less effective and efficient, especially the reviews of those PIAs that are complex in nature. It was noted that of the high priority files analyzed as part of the audit, the average length of time it took a recommendation letter to go through the quality assurance and approval process was 60 days (from the time a Review Officer provided the file to a PIA Manager until a recommendation letter was sent to the federal government institution). From a knowledge management perspective, not having sufficient information on file may make it difficult to leverage the analysis done on one PIA for subsequent PIAs that are related or similar in nature, especially when staff changes occur.

Recommendation #1

Formalize guidelines for the information that should be included in PIA files, focused on those core inputs that were critical to the analysis and development of the recommendations letter. For example, input from other OPC branches or correspondence/meetings with government departments or other external stakeholders should be included in the file. A checklist or other mechanism allowing for quick summation should be used to allow someone who is reviewing the file to understand the inputs and actions taken related to the PIA review. It is expected that more information would be included in the files of complex PIAs.

Management Response and Action Plan Responsibility / Deadlines
We agree with the recommendation. A checklist will be developed at the PIA Planning Retreat and implemented as part of our PIA review process. A&R – August 15th, 2012

Recommendation #2

Recommendation letters should be consistent in ‘look and feel’ and explicitly identify the actions the OPC are requesting of the government department.

Management Response and Action Plan Responsibility / Deadlines
We agree with this recommendation and strive to achieve a common look and feel. The letters of recommendation will be reviewed to ensure that actions requested by the OPC are explicitly identified. A&R – May 2012.

Finding 2: Current performance measurement should be improved

Performance measures for the review of PIAs have been set through the OPC performance management framework, and performance against these indicators is reported annually through the Department Performance Report (DPR).  The OPC performance management framework includes an indicator related to timeliness (percentage of PIAs completed within 120 days) as well as an indicator related to the effectiveness of PIA reviews in improving the privacy practices of government initiatives. The target set for this indicator is that 75 percent of recommendations lead to improved privacy practices. 

Related to timeliness, the majority of PIA reviews are not being completed in 120 days.  For example, of the 34 high priority PIAs in which the OPC sent recommendation letters to federal government institutions in calendar year 2011, the average time for the reviews was significantly more than 120 days (with some of that time attributable to external factors), and only four were completed within 120 days.  Timeliness is also measured through the monthly OPC scorecard.  The measures in the scorecard are activity driven (e.g., number of files received, processed, etc.) and treat all PIAs as equally important/relevant in the context of achievement of OPC priorities, regardless of their priority or complexity. 

Related to effectiveness, although there were more than 12 high priority PIA files reviewed in 2010-2011, the 2010-2011 DPR only measured effectiveness in the case of 12 PIAs.  An analysis of PIA files by the audit team indicated that follow up from the PIA review group with federal government institutions after a recommendation letter is sent is not being done, either to determine the extent to which recommendations were accepted, or to remind departments to provide the additional information that was requested (in the majority of recommendation letters, the OPC requests additional information from the federal government institution to better understand the management and safeguarding of personal information as outlined in the PIA). 

The Privacy Act Annual Report to Parliament also provides narrative description of the activities of the PIA review group.

Impact

If high priority PIAs are not measured separately, it is more difficult to understand how the PIA review process contributes to the fulfillment of the priorities of the OPC, and the value provided by the PIA review group. If follow-ups are not performed, the ability to measure the effectiveness of the PIA review process is limited.

Recommendation #3

Develop a formal process for follow-up with federal government institutions to understand the extent to which federal government institutions have implemented the recommendations provided by the OPC through the PIA review process.

Management Response and Action Plan Responsibility / Deadlines
We agree with this recommendation. A process for following up with institutions in currently in place, however, it has not consistently been applied. The process is now being monitored at the biweekly PIA meeting to ensure that it is being applied to all files in a consistent manner. A&R – already implemented.

Recommendation #4

Review the rationale for the performance measure related to the completion of PIA reviews in 120 days in light of current performance and the need to ensure the reviews of high priority PIAs are timely and relevant. Consider measuring the performance for high priority PIAs separately.

Management Response and Action Plan Responsibility / Deadlines
We agree with this recommendation. The performance measures will be reviewed at the PIA process planning retreat. We have previously experimented with measuring performance on high priority PIAs separately; however, this method did not provide a complete picture of the workload related to the reviews given to lower priority files, which do require resources even though they are not as in-depth. This created problems when planning workloads and assignments. We will revisit this decision at the PIA planning retreat. A&R in consultation with senior management – August 15th, 2012

Finding 3: Roles and responsibilities related to the PIA review process should be further defined throughout the OPC

The PIA review group regularly obtains input from other OPC branches to assist in their understanding of the issues related to the PIA they are reviewing, especially for complex PIAs.  Consultation is most often with technology (Technology Analysis Branch (TAB)) and policy (Legal Services, Policy and Research Branch), although the PIA review group does on occasion require legal advice form legal services as well.  There are no formal processes for obtaining this input, for example, related to:

  • considering if other OPC branches should be contacted;
  • contacting other OPC branches to obtain input; and,
  • for obtaining input from other OPC branches.

TAB has recently developed a process to support IT-oriented PIAs that are complex in nature, although this has not been formally defined or documented.   Through this process, TAB has indicated to the PIA review group that any request for support will be fulfilled within two weeks, either through actual advice related to a PIA Review (through a technical memo), or for PIAs requiring more effort, a ‘scoping document’ outlining the work required of TAB and estimated timelines.  

Branch plans and job descriptions of staff in other branches do not specifically identify their requirements to support the PIA review process.

Impact

Not having formal processes for collaboration between the PIA review group and other units within the OPC increases the risk that the PIA Review unit will experience long delays involving other OPC branches and will not be able to provide recommendations to submitting institutions in a timely manner, which in turn diminishes the ability of the OPC to influence the privacy practices of institutions.

Recommendation #5

Involvement of other OPC branches should be considered at triage, and should be added to the triage template.

Management Response and Action Plan Responsibility / Deadlines
We agree with this recommendation. The triage sheet was amended to include which branches should be consulted for the review. A&R – already completed

Recommendation #6

The responsibilities of other OPC branches related to the PIA review process should be defined, for example through Branch Plans or procedural documents.

Management Response and Action Plan Responsibility / Deadlines
We agree with this recommendation. The responsibilities of LSPR and TAB related to the PIA review process will be included in these branches’ Branch Business Plans as part of their ongoing activities. CSB with input from A&R, LSPR and TAB – June 2012

Appendix A - Interviewees

The following key individuals were interviewed as part of the audit process:

  • Assistant Commissioner;
  • Director General, Audit and Review;
  • Two PIA Review Managers;
  • PIA Review Officer;
  • Three representatives from other OPC branches (technology, policy, and legal) that provided input to the PIA review process; and,
  • Representatives from two federal government institutions that frequently submit PIAs to OPC and a representative from the TBS Information and Privacy Policy group.

Appendix B - Description of the PIA Review Process

The following process description provides an overview of the PIA review process, including those PIAs that are complex in nature information technology-oriented programs or activities:

Intake:

1) PIAs are received by the OPC and a standard letter is sent to the submitting institution informing the institution that the PIA has been received.

File Preparation:

2) A hardcopy docket is prepared for the file which assigns the PIA a file number, and includes the PIA, supporting documentation, and communications between OPC and the institution. Supporting documentation is also stored electronically in CCM Mercury organized by the same PIA file number.

Triage:

3) The PIAs that are received are assigned to a PIA Review Manager and triage is performed.

4) PIAs are triaged using the PIA Review Triage Template based on the following factors: alignment with the OPC’s four strategic policy priorities, public interest, parliamentary interest, number of people affected, and sensitivity of the issue. Based on the triage, the PIAs are assigned a priority level of Low, Medium, or High. PIAs that are determined to be high priority are communicated within the PIA ‘Files of Interest’ document which is communicated to the Privacy Working Group (PWG). Complex PIAs are most often determined to be high priority PIAs given the subject area.

5) PIA Review Managers assign PIAs to PIA Review Officers based on an officers’ availability as well as areas of expertise, as some officers have extensive knowledge in specific privacy issues (e.g. privacy issues related to health care organizations).

PIA Analysis:

6) PIA Reviews may be performed using the PIA Review template; however, experienced PIA Review Officers may choose not to use the template, although they would generally follow the same process. This would include determining if the Four Part Test has been fulfilled and conducting an overall assessment of the PIA. PIAs determined to be high priority are reviewed in greater detail.

7) During the PIA Review process, other OPC branches may be contacted to provide support to the review process, especially those PIAs that are complex in nature. Other OPC branches that are frequently contacted for completing PIA Reviews are the Technology Analysis Branch (TAB), and the Legal, Policy and Research Branch (primarily the Policy sub-function).

Recommendation Letter:

8) For PIAs that are determined to be low priority, a standard low priority letter is sent to the submitting institution identifying that the PIA was not subject to an in depth review, and that the OPC considers the file closed. The low priority PIA letter may include comments on the PIA advising the submitting institution to implement certain measures identified within the PIA.

9) For PIAs that are determined to be high priority, a letter is sent to the submitting institution identifying recommendations to the institutions, as well as requests for additional information to clarify certain issues, especially for those PIAs that are more complex in nature. The OPC provides a date for which the submitting institution should respond to the OPC’s recommendation letter.

10) The recommendations letter is formally reviewed and approved by one of the PIA Review Managers and the Director General, Audit and Review Branch. Based on the priority assigned to the PIA, the Assistant Commissioner may also review the recommendations before being provided to the submitting institution.

Follow Up Letter:

11) If no response is provided to the recommendation letter, a standard reminder letter may be sent to the submitting organization. If still no response is provided, a standard second follow up letter may be sent that notifies that the file will be closed indicating that non-responsiveness, and advising that OPC is an Officer of Parliament.

Date modified: