Nationwide Insurance offers financial services, property and casualty lines, and pet and travel insurance, among other services. At the 2021 DevOps Enterprise Summit, Ethan Culp (NETC Sr. Associate), Rusty Lewis (IT Audit Specialist), and Clarissa Lucas (IT Audit Director) described how the Nationwide Internal Audit office is putting DevOps theories into practice, how they think about risks and controls, and the influence of automation on risk mitigation.
As the Nationwide team shared, here are three lines of an organization. The first line owns risk and executes controls to manage them (e.g., development teams that write and develop code, and review and approve changes). The second line is responsible for policy creation, defining risk tolerance, and monitoring adherence to policies (e.g., an information risk-management function). The third line is Internal Audit, which provides assurance to the audit committee of the board and senior management through independent assessment of risks and controls.
Internal Audit achieves this by seeking to understand what could prevent the organization from achieving its objectives, taking into account all of the many risks. They evaluate the action management is taking to mitigate those risks within established tolerances and, to tie everything together, they conduct integrated audits to evaluate business versus controls and systems and applications that support business processes. They don’t tell management how to manage a particular risk, but they do partner with the first and second lines and external parties to provide assurance of the organization’s ability to meet objectives.
The primary risks of change processes are compromised systems, data availability, integrity, and confidentiality. Internal Audit performs audits to evaluate controls and determine whether the items are well managed, using a strong control environment to mitigate these risks. Currently, Internal Audit manages risk through testing that depends on three controls: approvals, segregation of duties, and metrics monitoring.
The objectives of the approvals control are that all application changes should be tested by a person other than the developer before they move into production, a sample of 1–30 changes are used for detailed testing, and documentation shows evidence of approval for sample changes at various stages throughout the change-management process.
Metrics monitoring includes reporting changes to monitor effects on business operations. New tools, analytics, and upskilling are allowing Nationwide’s Internal Audit team to test the full population of changes with greater accuracy, which enables continuous auditing in parts of the business and partnership between business units to mitigate risk as part of the development life cycle.
Segregation of duties in a DevOps environment means that duties are separated to ensure that the same individual isn’t writing, testing, and promoting their own code into production without independent checkpoints in order to guarantee that code isn’t malicious or fraudulent and that it doesn’t lead to system disruptions or data issues. To prevent these issues, the person writing the code reviews the code to detect defects early in the process and to maintain consistency in design and implementation, which allows for uniformity of understanding and helps team members be interchangeable. During automated reviews of tests, a script reviews a developer’s code and promotes it into production once it’s been tested, reviewed, and approved. This allows developers to more efficiently review code on demand while separating the duties between a human developer and digital worker.
In addition to segregation of duties, automated production deployment is also present in a DevOps auditing environment. In this process, the code is automatically moved to production as long as it meets predefined requirements, like a suite of controls, that lessen the risk that the code will not perform correctly or introduce vulnerabilities. Automated deployment control rejects code that doesn’t meet the predefined requirements.
Internal auditors test this control by looking at its configurations to determine whether it is designed to mitigate the risk by accepting only code that meets the requirements and rejecting code that does not. They also examine this control through observation and review once the control has already run to make sure code is appropriately accepted and rejected as expected. This is a different process than other audits, wherein auditors review many transactions, like code deployments or changes.
One example of how automated testing has been implemented effectively at Nationwide comes from one of the build teams, Partners and Beyond. This team is responsible for generating insurance quotes for customers and was previously a monolithic service hosted on premise. They are currently designing and building a new application using cloud-based managed ETL services and streaming data to themselves.
As the team develops a new process and skill-set, it is essential that they have breathing room to manage themselves. Partners and Beyond have developed a manager-approved deployment pipeline that goes through a series of checks and gates that allows them to operate more efficiently. This pipeline has multiple dependencies but can only advance if the automated testing gives the green light and if all events are logged in their tool. Errors trigger automated emails and are displayed in a process tree dashboard.
This automated testing also promotes poly-skilling through code reviews and an emphasis on documentation, which also helps the audits maintain objectivity. Developers are spending less time testing and are able to do more releases in a given time, so velocity goes up as lead time decreases. Faster deployments and more releases keep Nationwide competitive in their marketplace.
The Nationwide Internal Audit office is itself also adopting more Agile practices. In late 2020 and early 2021, the Internal Audit organization tested specific financial reporting controls in partnership with Nationwide’s external auditors in a high-profile endeavor with a tight timeline that drew attention from key stakeholders. Due to the tight timeline, the Internal Audit office needed to think about work differently, and because they were performing testing on behalf of another party, they needed to be able to pivot quickly and maintain continuous feedback. To do so, they incorporated Agile practices.
The self-organized auditing team avoided multitasking using dedicated resources, which differs from the usual operating model, where an auditor might have multiple engagements happening at the same time. Objectives were clear, so minimal time was spent negotiating priorities. The project manager determined what testing was to be done and which procedures to perform, while the team was responsible for deciding how best to accomplish that testing. This flexibility of resources let people jump from one task to another as they were finished without having to be formally assigned items by the project manager.
The team incorporated continuous delivery in short sprints by dividing testing into four buckets with a number of controls to be completed in each bucket. Each bucket lasted approximately one month. They worked together frequently with stakeholders throughout the audit, which was accomplished via iterative meetings with internal and external stakeholders, after which the team sent out detailed notes and sought feedback—a step toward adoption of the Agile principle in which business people and developers work together daily throughout the project.
This team also conducted daily standups and reinforced the urgency of the audit, which allowed them to shift resources to items that needed attention, to share knowledge, and to change testing procedures as needed. A blameless retrospective review was held at the end of the third bucket to reflect on opportunities to be more effective during the final bucket, during which this team found a way to automate testing of controls in the R programming language to eliminate the need to manually filter via Excel, saving fifteen hours per quarter.
In a future auditing pilot, Nationwide will add more DevOps and Agile practices by prioritizing customer needs via aligning Audit with the client’s workflow process; increasing focus on continuous delivery through sprints and fostering a collaborative environment; continuing to explore ways to automate tests; and holding a blameless post mortem alongside clients.