Audit of Applied Research

This page has been archived on the Web

Information identified as archived is provided for reference, research or recordkeeping purposes. It is not subject to the Government of Canada Web Standards and has not been altered or updated since it was archived. Please contact us to request a format other than those available.

Office of the Privacy Commissioner of Canada

March 2011


1. Executive Summary

1.1 Background and Context

The Office of the Privacy Commissioner of Canada (OPC) is an Agent of Parliament mandated to oversee compliance with both the Privacy Act, which covers the personal information-handling practices of federal government departments and agencies, and the Personal Information Protection and Electronic Documents Act (PIPEDA), Canada's private-sector privacy law.

As outlined in the OPC’s 2010-2011 Report on Plans and Priorities (RPP), OPC’s strategic outcome is to ensure that the privacy rights of individuals are protected.

To achieve this strategic outcome and fully utilize the powers at her disposal, the Commissioner structured Research and Policy Development as one of the four areas of OPC’s activities. Applied research-related activities at the OPC include:

  • Researching trends and technological developments;
  • Monitoring legislative and regulatory initiatives;
  • Providing legal, policy, and technical analyses on key issues to support the development of policy positions that advance the protection of privacy rights; and,
  • Supporting the Commissioner and senior officials in providing advice to Parliament on potential privacy implications of proposed legislation, government programs, and private-sector initiatives.

The Research, Education and Outreach (REO) Branch and the Legal Services, Policy and Parliamentary Affairs (LSPPA) Branch perform the majority of the above-mentioned applied research activities at OPC, either internally or through research contracts with external contractors. While the REO Branch conducts applied research as a discrete activity, the LSPPA Branch conducts applied research as an integral part of its other activities, such as policy advice-related activities.

The audit did not include the Contribution Program within the REO Branch, which provides funding for research and public education activities on commercial privacy issues by academics and not-for-profit organizations.

The OPC also identified four top strategic policy priorities to focus its approach to emerging privacy issues over the coming years. These four top strategic policy priorities are:

  • Information Technology;
  • National Security;
  • Identity Integrity and Protection / Identity Theft; and,
  • Genetic Information.

The purpose of this audit is to provide assurance on the effectiveness of risk management, controls, and governance processes that support the applied research used to: inform senior management communications; forecast the proactive identification of privacy issues; support policy development; and, provide policy advice to Parliament and Parliamentarians.

1.2 Summary of Observations

The key observations with regards to Applied Research are provided below.

Strengths

  • Employees of the Research, Education and Outreach (REO) and Legal Services, Policy and Parliamentary Affairs (LSPPA) Branches receive adequate training and are provided with tools and templates to support ongoing research-related activities;
  • Results of applied research projects contracted outside OPC are communicated through the OPC website, workshops and panel discussions opened to the public;
  • Applied research projects conducted by external contractors are formally managed by the REO Branch; and,
  • Based on interviews with selected OPC staff, internal research activities are responsive to the organization’s needs.

Findings

  • There is no overarching plan to coordinate discretionary research activities within the OPC;
  • Roles, responsibilities and processes to identify, prioritize, assign and oversee internal research activities within the REO Branch are informal; and,
  • Performance indicators related to applied research are subjective and are difficult to measure consistently.

1.3 Conclusion

Based on the aforementioned observations and overall scope of the audit, the OPC has moderate issues related to the effectiveness of its current risk management, controls, and governance processes that support applied research. The recommendations included in this report are intended to strengthen the risk management, controls, and governance processes that support the applied research. Management responses are included at the end of each finding.

Based on our professional judgment as auditors, sufficient and appropriate audit procedures have been conducted in accordance with the Treasury Board (TB) Policy on Internal Audit, and evidence gathered supports the accuracy of the conclusions contained in this report. The conclusion is based on a comparison of the conditions, as they existed at the time, against pre-established audit criteria that were agreed to with management. The evidence has been gathered to provide senior management with reasonable assurance of the accuracy of the conclusions drawn from this audit. This report and audit were conducted for OPC management purposes. Use of this report for other purposes may not be appropriate.

2. Audit Objective, Scope and Approach

2.1 Background

Applied research-related activities at the OPC include:

  • Researching trends and technological developments;
  • Monitoring legislative and regulatory initiatives;
  • Providing legal, policy, and technical analyses on key issues to support the development of policy positions that advance the protection of privacy rights; and,
  • Supporting the Commissioner and senior officials in providing advice to Parliament on potential privacy implications of proposed legislation, government programs, and private-sector initiatives.

The Research, Education and Outreach (REO) Branch and the Legal Services, Policy and Parliamentary Affairs (LSPPA) Branch perform the majority of the above-mentioned applied research activities at OPC, which account for approximately $2M in salaries and operations & maintenance expenses.

The REO Branch has approximately ten full-time equivalents (FTEs) within the Branch, all of whom report to the Director of Research, Education and Outreach. In addition to its research responsibilities, a significant proportion of REO Branch staff time is allocated to assisting other Branches, external organizations and federal departments in their privacy-related work, including:

  • fulfilling OPC commitments to the International Organization for Standardization (ISO),Interpol and the Organization for Economic Co-operation and Development (OECD);
  • working on specific OPC investigations;
  • assisting in the establishment of the Toronto office and providing research support to the Assistant Commissioner, PIPEDA;
  • organizing consultation events, leading two of the four priority working groups, and organizing the Commissioner’s policy-making working group, known as Privacy Working Group.

Applied research within the REO Branch is prioritized on a multi-term basis. Policy priority working groups (which include individuals from the Operational Branches, Policy Branches, and Research Branches at OPC) each have three-year work plans which outline intended research subjects and activities. REO helps identify these areas for research, as well as medium and long-term research initiatives to support the Commissioner’s identified policy priorities.

The Policy and Parliamentary Affairs unit, which is part of LSPPA, supports internal research at OPC mainly through timely research performed to monitor legislative and government program initiatives. The unit has six full time policy and parliamentary affairs positions with the Director of Policy and Parliamentary Affairs and the Senior Strategic Policy Advisor reporting to the General Counsel. Their role is primarily to provide policy advice, with research being but one element of the policy development process. Based on the analysis of their research, the Branch provides advice and briefings to the Commissioner and senior management on the appropriate policy position the OPC should take in order to protect and advance privacy rights in Canada. The applied research the LSPPA Branch performs helps prepare and support OPC senior staff and the Commissioner/Assistant Commissioners in appearances before Parliamentary Committees and in relations with Parliamentarians.

2.2 Audit Objective

The purpose of this audit engagement is to provide assurance on the effectiveness of risk management, controls, and governance processes that support the I&I Branch in responding to inquiries from the public. Detailed audit criteria can be found in Appendix B.

2.3 Audit Scope

For the purposes of this audit, applied research is defined as the research, which OPC conducts, uses, and applies internally, to: support senior management communications; forecast the proactive identification of privacy issues; support policy development; and, provide policy advice to Parliament and Parliamentarians.

The audit did not include the Contribution Program within the REO Branch, which provides funding for research and public education activities on commercial privacy issues by academics and not-for-profit organizations. An evaluation was recently completed for the Contribution Program in 2009, which led to a renewal of the Program’s terms and conditions.

2.4 Audit Approach

The approach and methodology used for this audit is consistent with the Internal Audit standards as outlined by the Institute of Internal Auditors, and is aligned with the Internal Audit Policy for the Government of Canada.

As an Agent of Parliament, OPC works independently from the Government of Canada and is therefore not obligated to follow the management improvement initiatives put forward in the Federal Public Service.

Nevertheless, the OPC strives to maintain a control framework for Applied Research that is reflective of industry leading practices. Consequently, the framework of Core Management Controls and Audit Criteria (CMC) established by the Office of the Comptroller General of Canada (OCG) and the Management Accountability Framework round VII criteria (MAF VII) were leveraged to develop the audit criteria detailed in Appendix B. Other criteria were also included to ensure appropriate coverage of the aforementioned audit scope and identified risks.

Based on risks identified throughout the planning phase of the audit, a risk-based audit program was developed to provide more details on how the various audit criteria and risks were addressed. The audit program includes the following audit procedures:

  • Review of REO and LSPPA Business Plans and Applied Research Plans
  • Review of working group terms of reference, meeting minutes, and numerous other supporting documents, such as work plans;
  • Review of a sample of research contracts, reports, briefing notes, etc.;
  • Interviews with individuals from across the OPC. The list of interviewees can be found in Appendix A.

The audit was conducted within the following timelines:

  • Planning Phase : September 2010 – October 2010
  • Examination Phase: October 2010 – December 2010
  • Reporting Phase: December 2010 – March 2011
  • Presentation to the OPC Audit Committee: March 2011

3. Findings and Recommendations

3.1 Strengths Noted

The following strengths were noted with regards to the current approach to applied research:

  • Employees of the Research, Education and Outreach (REO) and Legal Services, Policy and Parliamentary Affairs (LSPPA) Branches receive the necessary training and are provided with tools and templates to support ongoing research-related activities;
  • Results of applied research projects contracted outside OPC are communicated through the OPC website, workshops and panel discussions opened to the public;
  • Applied research projects conducted by external contractors are formally managed by the REO Branch; and,
  • Based on interviews with selected OPC staff, internal research activities are responsive to the organization’s needs.

3.2 Audit Findings

3.2.1 Lack of Overarching Plan for Applied Research

Although applied research activities are included at a high level in business plans and in the work plans of the four policy priority working groups, there is no overarching plan to coordinate discretionary research activities within the OPC.

The OPC Research and Policy Development program activity identified in the OPC Report on Plans and Priorities (RPP) encompasses a variety of work, including researching trends and technological developments, monitoring legislative and regulatory initiatives, providing legal, policy, and technical analyses on key issues. These activities support the development of policy positions that advance the protection of privacy rights, as well as supporting the Commissioner and senior officials in providing advice to Parliament on potential privacy implications of proposed legislation, government programs, and private-sector initiatives. The REO Branch and the LSPPA Branch perform the majority of these research activities.

Applied research within the REO Branch is discretionary in nature, and centered on four policy priority working groups (National Security, Genetic Information, Information Technology, and Identity Integrity) which include individuals from the operational branches, policy branches, and research branches at OPC. Each of these working groups has three-year work plans, which outline intended research subjects and activities. The REO Branch helps identify these areas for research to support the Commissioner’s identified policy priorities.

The LSPPA Branch conducts internal research in support of its primary responsibilities to monitor legislative and government program initiatives and to provide advice and briefings to the Commissioner and senior management on the appropriate policy position to take in order to protect and advance privacy rights in Canada. The research that LSPPA performs is typically less discretionary in nature given the need to respond to short term demands driven by external factors.

Some discretionary research objectives and activities are currently documented in several different documents, such as the REO Business Plans, the working group performance indicators (which are essentially the work plans of the working groups), the OPC Report on Plans and Priorities and a REO working document (Notes on Identifying REO Priorities). These objectives are, however, not consolidated in one comprehensive and overarching applied research plan that provides direction with respect to discretionary applied research priorities, roles and responsibilities. By their nature, discretionary research activities require more upfront planning to ensure they meet the needs of the organization. There is also no documented analysis of OPC’s overall research capability and capacity centered on the four policy priorities; research activities currently occur primarily within the Information Technology priority.

As mentioned above, the REO Branch relies on the four policy priority working groups to drive internal research activities. These activities are documented in the working groups’ performance indicators (i.e. work plans). Of the four working groups, only the Information Technologies working group has a detailed research plan. Also, only the Information Technologies and the National Security working groups maintain minutes for their regular meetings; during the audit period, less evidence was available from the other working groups to help determine how they were active in, amongst other things, driving research activities.

Impact and Risk Level

The lack of an overall plan for discretionary research activities increases the risk that these activities are not appropriately coordinated, that applied research resources are not used as efficiently and effectively as possible, and that some research efforts are duplicated.

The lack of an analysis of research capability and capacity increases the risk that gaps of coverage related to the four policy priorities will not be identified and corrective measure will not be implemented in a timely manner.

Recommendation #1

Develop an OPC-wide applied research plan in order to identify and coordinate discretionary applied research priorities and activities (e.g. trends and new technical developments), and track progress against the plan on a regular basis.

Management Response and Action Plan Responsibility / Deadlines
1. Working in conjunction with the heads of the four Policy Priority Working Groups and the Privacy Working Group, the Research, Education and Outreach Branch will collect current work plans and assess the research necessary to meet work plan objectives.

Research, Education and Outreach Branch will also work with Branch heads to address research needs arising out of Branch Business Plans.

Research, Education and Outreach Branch will then develop an OPC-wide research plan, and consult with the Privacy Working Group to ensure there are no omissions or overstatements.

Monitoring will be incorporated into the regular meetings of the Privacy Working Group.

This assessment will take place in April and early May, in order to incorporate the business planning process.

Specific and discrete research activities needed to inform legal opinions, policy development and parliamentary submissions are distinguishable as they are typically less discretionary in nature, must be carried out in response to external stakeholders, are driven by external factors and demands, are time sensitive, and cannot be planned, coordinated or driven in the same way.
1. Research, Education and Outreach Branch

Annually, in May

Recommendation #2

Formally analyze the research capabilities and capacities of the organization based on the four policy priorities and implement corrective measure as needed.

Management Response and Action Plan Responsibility / Deadlines
The Research, Education and Outreach Branch will address this recommendation in two ways: Research, Education and Outreach Branch
1. The reassignment of portfolios in conjunction with the arrival of several new research analysts, thereby closely matching portfolios to policy priorities; and 1. Reassignment of portfolios

April 2011
2. The introduction of an occasional research training program, bringing in experienced research managers from the Government of Canada and elsewhere to discuss comparable administrative and research frameworks. 2. Occasional research training program

May 2011 and onwards

3.2.2 Lack of Formal Processes for Internal Research

Based on interviews with selected OPC staff, internal research activities within the REO Branch are responsive to the organization’s needs; however, roles and responsibilities have not been clearly defined, and processes to identify, prioritize, assign and oversee internal research activities within REO are informal.

Research activities conducted internally by employees of the LSPPA Branch in support of their legislative monitoring and policy development responsibilities are formally reviewed and approved by LSPPA management. Externally contracted research projects for the branch are identified in the LSPPA business plan, and specific resources are assigned to the projects. Research activities in LSPPA, however, are not systematically shared or disseminated to other OPC branches (it should be noted that the dissemination of policy advice based on this research was not within the scope of the audit). A pilot project using SharePoint technology and the Legal and Policy Corner initiative were under development at the time of the audit, and are intended to further facilitate the sharing of LSPPA research results with all OPC staff.

Applied research projects conducted by external contractors are formally managed by the REO Branch; however, internal research activities of the REO Branch are not as formally supervised and approved. The research project management approach is described as informal; processes to identify, prioritize, assign and oversee internal research activities within the REO Branch are informal. Consequently, staff members of the REO Branch respond directly to requests coming from other branches without having to go through a challenge function to determine which requests should be considered as priorities, or if there is overlap with other internal research activities being conducted. In addition, time and task allocation decisions are made at an individual level, and do not necessarily consider costs vs. benefits.

Impact and Risk Level

The informal roles, responsibilities and processes to manage internal research activities within REO increase the risk that resource allocation decisions made individually by REO staff may not be appropriate and/or adequately consider costs vs. benefits, including an assessment of alignment with REO and OPC objectives. The informal roles, responsibilities and processes also increase the risk of gaps and overlaps between the activities of all research staff members.

The informal sharing of the output of REO and LSPPA internal research activities increases the risk that other OPC branches may duplicate and/or not fully leverage information stemming from research activities.

Recommendation #3

Proceed with improving systems functionality to support the capture and reporting of more useful information, as well as implementing the newly developed taxonomy for inquiries.

Management Response and Action Plan Responsibility / Deadlines
1. The creation of a new position of Manager, Research, means direct management control of this function is now in place.

Tasks are being assigned and projects monitored using SharePoint platform tools, leading to greater monitoring and control over the internal research function.
Research, Education and Outreach Branch

March 31, 2011

Recommendation #4

Consider the active use of shared folders or tools such as EDRMS and the full implementation of the SharePoint pilot project to provide open access to all OPC staff and disseminate information on REO and LSPPA internal research activities to other OPC branches.

Management Response and Action Plan Responsibility / Deadlines
1. The Research, Education and Outreach Branch began using SharePoint to coordinate work during the planning and implementation of the 2010 National Consultation Sessions.

The Branch has moved to incorporate most of its work on a dedicated SharePoint site, including a listing of current research work.

As current research projects are completed, they will be assessed for the possibility of wide distribution through the office.

In many cases, research commissioned or created by the Office is incorporated into investigative reports, audit reports, internet fact sheets, annual reports , blog posts and other materials that are made generally available to staff and the public.

In addition to other dissemination vehicles used through OPC and on the OPC website, LSPPA has developed and implemented a Parliamentary Affairs SharePoint site (PIMS), and is developing a Legal SharePoint site. As well, it has launched a “Legal and Policy Corner” to further disseminate Branch knowledge with other Branches of OPC.
1. Research, Education and Outreach Branch

March 31, 2011

3.2.3 Lack of Objectively Measurable Performance Standards

Although performance indicators related to applied research have been defined in the OPC Report on Plans and Priorities and Branch Business Plans, the indicators are subjective and are difficult to measure consistently.

The Report on Plans and Priorities and the REO and LSPPA Branch business plans provide three performance indicators for research and policy activities (program activity 2) conducted internally at the OPC:

  • 75% effectiveness in adding value to the public and private-sector stakeholders through the OPC information and advice on their policies and initiatives.
  • 75% effectiveness in adding value to Parliamentarians from the OPC views on relevant laws and regulations.
  • Initiatives under all four (100%) OPC priority privacy issues have involved relevant stakeholders and there is documented evidence demonstrating that they were impacted by the OPC research products and outreach materials.

These three performance indicators are difficult to measure objectively and consistently, in particular with respect to the applied research activities of the REO Branch. As indicated in the Implementation Strategy for the OPC Performance Measurement Framework 2010-2011, it is currently contemplated that these indicators will be measured through self-assessments. Formal measurement of the performance indicators has not yet occurred.

Impact and Risk Level

The lack of objectively measurable performance indicators increases the risk that the performance of research activities is not being adequately measured, and that performance corrective measures may not be identified and implemented in a timely manner.

Recommendation #5

Refine current performance measures to ensure that they are objectively measurable, in particular with respect to the applied research activities of the REO Branch. Finally, these performance measures should be formally monitored on a regular basis, reported to senior management and corrective measures should be identified and implemented as necessary.

Management Response and Action Plan Responsibility / Deadlines
1. Performance measures are being examined as part of the revision of the OPC internal scorecard.

As well, the creation and staffing of the position of Manager, Research is applying another level of detailed assessment and tracking to the research process within REO
1. Performance measures – April 2011

Management supervision – ongoing

Appendix A – Interviewees

The following key individuals were interviewed as part of the audit process:

  • Director, IM/IT
  • Assistant Commissioner
  • Director General, Investigations and Inquiries
  • Director, Financial and Administrative Services
  • Director, Policy and Parliamentary Affairs
  • Director, Communications
  • Complaints Registrar
  • Senior Security and Technology Advisor, Research, Education and Outreach
  • General Counsel, Legal Services, Policy and Parliamentary Affairs
  • Director, Research, Education and Outreach
  • Senior Research Analyst, Research, Education and Outreach
  • Director, Human Resources
  • Director General, Audit and Review
  • Director General, Corporate Services
  • Commissioner
  • Research Analyst, Research, Education and Outreach
  • Senior Strategic Financial Officer

Appendix B – Audit Criteria

  Criteria
1.1 Operational plans and objectives have been established related to applied research and linked to the mandate and identified priorities of the OPC.
1.2 Roles and responsibilities for applied research have been defined and communicated.
1.3 The organization structure for applied research is appropriate and conducive to the achievement of OPC objectives.
2.1 A comprehensive policy framework has been established for the applied research process, and is supported by appropriate tools and a training and awareness program.
2.2 A program and project management framework for the management of all applied research projects is established. The framework ensures the effective execution of all projects. Project closure procedures include evaluation of realized benefits.
2.3 Results of research projects are appropriately disseminated.
3.1 Applied research budgets are appropriately developed, approved and monitored.
3.2 An applied research performance monitoring process is in place to evaluate applied research service delivery, and monitor applied research contributions to achieving OPC objectives.
Report a problem or mistake on this page
Please select all that apply (required): Error 1: This field is required.

Note

Date modified: