Written by: David Mercieca, MS Practice Lead
In the last decade, technology and engineering practices have changed dramatically. The introduction of cloud technology has made a significant impact on the ability to deploy and test software and infrastructure platforms at scale.
With the introduction of “Everything as Code”, or GitOps, the importance of developing a clean, stable and well documented pipeline was for many the obvious choice when it came to release management. However, it is still fairly common to find traditional approaches to software and infrastructure management that are complex, brittle and a maintenance minefield.
As a software engineer I have experienced several iterations of software deployment practice, from MSI builds and DLL hell to manual handover of build artefacts to scripting and part “automation”. On more than one occasion, I would start on a new development team and notice deployment times were taking days and when I asked the question “why is this taking so long?” the response was usually “this is great – it used to take weeks”. Distributed teams also started to appear making the software development life cycle even more complex to manage. The overhead of co-ordinating deployments across geographical regions exacerbated an already convoluted and time consuming process.
Then, along came Continuous Integration / Continuous Delivery (CICD). Could this be the answer to the problem? Yes and No. The No stems from bad practice that in many cases, organically led to what I refer to as “pipeline sprawl”. Several hundred duplicated pipelines coupled with scores of custom build servers sitting under desks would appear over time. The cost and risk to the organisation became so high that decommissioning what they had invested heavily in over several years and starting over was the only viable solution.
Patterns and Practices started to develop in line with the demand for faster deployment cycles. The introduction of Agile spawned a new set of practices that led to DevOps. The cornerstone to DevOps was CICD and a structured, flexible approach to managing the Software Development Lifecycle.
Common Release Antipatterns
Deploying software manually and deploying software “late” in the cycle (i.e deploying to a production-like environment at the end of the development cycle) were two fundamental anti-patterns that were identified early as the first targets for overhaul. A distinct lack of key infrastructure available to on-premise projects coupled with waterfall style approaches to development were the perfect storm that had so many large projects in trouble early on. Multiple streams of work consumed all the testing resources leaving developers to test on laptops, compromising integration testing cycles. The introduction of the Cloud did alleviate the lack of resources but without a more holistic change to the ways of working, problems still occurred on a regular basis.
Fear of Change
One of the biggest challenges was with traditional operations teams. The same teams that had heavily customised “pet” servers sitting under their desks and wrote huge documents to describe a release.
When talking with operations teams, I came to understand the basis for their fears was risk. Their fears were reasonable – in many organisations, developers were the only technical staff that understood pipelines and most had full access to all pipelines including production. The answer to the problem was adherence to a traditional and critical operational control, Segregation of Duties.
If it could be demonstrated that segregation of duties was feasible with an automated pipeline, there would be less reason for operations teams to reject these new methods.
Upon demonstration, these teams realised that they could save days, if not weeks of manual environment management by running automated pipelines.
The approach described below includes two key elements to support segregation of duties:
- Role based authentication: only those users with access to an environment can view or execute pipelines.
- Where necessary, in larger enterprises, web hooks are integrated into the upstream pipeline to ensure formal change control is in place for each production deployment.
Maturing Release Patterns
The Fan-In / Fan-Out Pattern
Componentization became popular in order to better support Service Orientated Architectures, the precursor to modern Microservice Architecture. So how could changes to components and libraries developed in-house be managed as the system matured? Several patterns were developed including the Fan-In / Fan-Out Pattern.
The Fan-In / Fan Out Pattern provides an end-to-end pipeline that ensures that any change in a dependent library triggers a re-build of the downstream component(s) that rely on it.
This “fail fast” approach to building software, ensures that all dependencies are current with every “check-in”. This approach requires discipline from the development team and adherence to maintaining comprehensive test suites in line with the platforms’ functional and nonfunctional requirements.
The Immutable Release Pattern
The key construct behind immutable deployments is to build the components to an exact set of specifications without intervention or manipulation. In this case what the software vendor builds and releases is shipped and packaged to a base set of agreed specifications. This also applies to the infrastructure supporting the application. A change to the specification triggers a release, replacing the current environment state.
The Managed Service Provider (MSP) has a wide spectrum of customer environments they manage. Standardisation through automation and an “everything as code” approach is key to the success of any modern MSP. Leveraging patterns such as the Immutable Release Pattern enhances the MSP’s ability to respond to changes quickly whilst documenting each change as part of the build process itself.
Large enterprise and government customers often rely on third party software vendors to develop, enhance and maintain their custom application code but require a more modern and less disjointed approach to release management. The diagram below describes a more complex and less well known approach to ensuring a fully integrated solution to deploying software and infrastructure using automation.
In this example, the software vendor has full autonomy to run their software build process to suit their needs. The build process leverages a public repository (in this case it’s Maven Repo) via a proxy in the MSP’s management account to pull down any 3rd party dependencies. IAM role based access to the management account is timed via an STS call to ensure adherence to the principle of least privilege. On successful completion of the build phase, the artefact is published to the MSP’s repository, in this case AWS Code Artifact. Amazon EventBridge polls AWS CodeArtifact and when a change is detected, triggers a pipeline that bakes the latest version of the application binary to an EC2 Image using Amazon EC2 Image Builder and Cloud Formation.
Several deployment options can be chosen including Blue / Green, Rolling and Canary to support the required use case. In this example the pipeline applies a second CloudFormation template. This template performs a rolling deployment using an EC2 Auto Scaling Group. A rolling deployment removes an old instance only after creating a new instance to avoid downtime. This template is decoupled from the previous CloudFormation template to make the solution modular whilst only deploying the minimum state change to the customer environment.
In conclusion, automation is key to a successful release management strategy but proceed with care. A well thought out and flexible design is crucial to managing an ever changing landscape. A well designed pipeline is self-documenting and applies all the principles of peer review, version control and rollback of not only the software it supports but its own metadata.
