Has Validation In An Agile Software Development Environment Come Of Age?
Contributed Commentary by Hugh O’Neill
December 12, 2019 | Although Agile software development originated in the 70s (probably earlier), I remember it becoming a topic of interest in Pharma IT circles in the late 1990s / early 2000s, around the time that the Agile Manifesto was published. The concept of flexibility and quick feedback was very exciting and made perfect sense, but the lack of formality and documentation made it seem incompatible with regulated environments. Today it is accepted that agile software development can work well within a formal project and most software vendors supplying the pharmaceutical industry have an agile aspect to their Software Development Life Cycle (SDLC).
I believe that Continuous Validation (CV) SDLCs and Continuous Delivery (CD) are particularly suitable for validated applications in regulated industries. Most approaches that I have seen as a customer have been a hybrid of traditional validation approaches coupled with an agile software development process. This allows the development team to get rapid feedback from the product managers but still requires a lengthy validation phase when a new release is delivered to the customer. At PHARMASEAL, we have developed a CV process that has the quick feedback loop and course correction aspects that makes agile so attractive but also maintains, in real-time, the control and traceability that is required by validation. It’s possible to react quickly to changing requirements and efficiently deliver quality software to the customer.
We are not alone; other companies are using similar CV SDLCs. Furthermore, the principles of CD, upon which these processes are based, are becoming established in other regulated environments, such as banking, and are well suited to these environments.
At PHARMASEAL, our SDLC consists of a continuous cycle of discrete steps, with one iteration for each feature implemented.
Requirements are captured in a requirements management tool. All work on the software (features, bugs and housekeeping) goes into this tool as small chunks of work. Prioritised lists can be generated alongside release markers to manage application development. All components are coded and managed in a version control tool (the source code, test code, build scripts, etc). The tool maintains a complete history of the application and allows you to roll back to any previous state.
Each new feature added to the system has a set of automated tests. These form our regression test suite, which is run for every change to the application. Each test is tagged with the requirement number to maintain traceability. Automated checks also scan the source code for security vulnerabilities and to confirm the code meets coding standards. After the automated tests, a mandatory independent Peer Review ensures that the change meets quality requirements for design, implementation, and testing. The final step is acceptance where the Product Owner confirms the change satisfies the requirement. If any changes are required, the developer must submit them through a repeat of the above process.
The whole cycle is managed with an integrated set of tools, all with user authentication and audit trails, so every activity is recorded against who did it and when. Once a change has been committed and accepted, it is considered done and ready for deployment but is held until the next planned release.
The PHARMASEAL release process
TrialGrid describes a similar CV process in recent blogs (Continuous Validation and Tracing Bug Fixes). Their process focuses on version control to maintain all application assets and manage the continuous validation pipeline. The requirements (and bugs) are tracked in an Issue list with unique identifiers referenced in the commit messages, and tagged in the testing, to enable traceability.
Every component is under version control and as each change is committed, the application is automatically built in the Dev environment for review. An automated pipeline of around 40 tests and other quality checks are run and all must pass for the process to continue. A peer review is required for the change to be accepted and this process is automatically recorded by the version control system with comments and responses tracked along with who did what and when.
The commit process also includes creation of an integrated hyperlinked validation package, generated automatically from the artefacts, workflow, and audit trail data held in the system. This package organizes the validation evidence so that an audit can be remote and largely self-guided.
Each successful change is delivered immediately to the Beta environment. Periodically, at times agreed with their customers, the accumulated changes are promoted to Production. For production releases, there is an additional manual review and the validation package is signed and retained.
Continuous Delivery ~ Continuous Compliance
Both the PHARMASEAL and TrialGrid processes are based on the principles of Continuous Delivery. Dave Farley, author and speaker, has recently posted a blog describing how CD satisfies the key requirements of regulatory compliance. He comments:
“when describing the CD approach to people I am regularly asked, “Yes, that all sounds very good, but it can’t possibly work in a regulated environment can it?”. I have come to the opposite conclusion. I believe that CD is an essential component of ANY regulated approach.”
The features of CD that make it ideal for a validated environment are:
- Everything is kept under version control
- Automation of repetitive tasks
- Continuous integration and continuous testing
- Build quality in
CD has been scientifically shown to be linked to higher software delivery performance (Accelerate: The Science behind DevOps: Building And Scaling High Performing Technology Organizations, by Nicole Forsgren). From a validation perspective, it is predictive of specific measures of quality, lower change fail rates and lower levels of rework or unplanned work (time spent on break-fix, emergency software deployments and patches)—the latter in a statistically significant way. I think this demonstrates that CD does lead to higher quality software development.
Software in the pharmaceutical industry has always been conservatively managed. It is not surprising that agile methodologies would be slowly and gradually adopted, especially when the Agile Manifesto plays down the value of documentation and process which are the core of validation. The significance of CD is that it emphasizes technical practices which bring greater capability and control. These practices do not conflict with the more ceremonial team and management practices of other agile methodologies. Indeed they are complementary and we have implemented the technical practices of CD along with sprints, stand-ups, and retrospectives taken from Scrum.
The principles of CD, together with the availability of easily implemented, powerful tools to support it, provide a secure framework that enables truly agile software delivery whilst genuinely enhancing quality and control. CD promotes fast feedback, while the work is still fresh in everyone’s mind, so fixes and course alterations are quicker and easier. The traditional methods of testing and carrying out security reviews at the end of a development cycle puts these steps on the critical path, every fix required slows down delivery and, to compound things, the issues have been given time to become embedded, making them more difficult to fix. Another big benefit of the continuous validation approach is that every change, even an emergency hotfix that needs to be deployed to production that night, is of the same quality. If you rely on manual testing and documentation you are forced to make compromises in these situations.
I have no doubt that we will see more examples of SDLCs featuring these processes soon, and customers, suppliers and the regulatory authorities will all benefit from the increased efficiency and control that they provide. It will take some time for vendors with established products to make the transition, as the changes are fundamental, but the benefits are clear and the decision will be more about how and when, rather than why. Agile software development in a validated environment has come of age.
Hugh O’Neill is Director of Operations and Quality at PHARMASEAL International Ltd. Hugh is a PhD scientist with more than 30 years’ experience in the pharmaceutical industry, much of it with IT responsibility. He can be reached at hugh.oneill@pharmaseal.co.