Today, I will show how we do continuous delivery with the new ANZ Plus app, and how we go from developers opening a pull request to having the app out in customers’ hands in five days.
ANZ Plus is a mobile-first product focused on customer financial well-being. It is the core product around which ANZ is transforming into a modern digital organisation and has all kinds of exciting features, both from a technical and a user perspective. To see it for yourself, you can find the app at anz.com.au/plus.
As the founding mobile development partner for ANZ Plus, Itty Bitty Apps has worked closely with the ANZ team for the past two and a half years, enabling them to establish and grow the ANZ Plus mobile team, which now exceeds 100 engineers & designers across iOS and Android.
If you are interested in learning more about the architectural principles that underpin the app, such as offline data caching with idempotent synchronisation, unidirectional data flow, and eventual consistency, I presented on the technical underpinnings of the app at GDG Melbourne.
Since that presentation, we’ve hit some exciting milestones. The ANZ Plus application now has over 500 million dollars in funds under management and over 40,000 customers, with thousands more joining each week. We’ve also released new features, such as increased payment limits, Google pay integration, BPAY, PayID, and dynamic CCVs. As you can see, delivery is rapid.
On average, the time between a developer raising a pull request until it’s merged to the main branch is 2.5 days, and the average time from merging a pull request until it’s out in a customer’s hands is three days.
Both the iOS and Android versions of ANZ Plus are released as updates into App stores twice a week, and so far, we’ve had over 300 releases of the Android and iOS apps.
When you add that all together, the average cycle time on a pull request is five business days from a developer raising it until it’s out in a customer’s hands on their mobile device, helping them manage their money.
Move fast, break fix things
We didn’t begin continuous delivery with these specific numbers as a benchmark of how quickly to release. Our initial goal was modest – to deploy the app once a fortnight. After each release, we looked at what broke in our process, and we’d fix that up. Every improvement gave us room to go a little faster, so we tried to speed up again to see what broke next. We iterated and improved by fixing what slowed us down.
Continuous Delivery Pillars
We have four underlying pillars of continuous delivery. These are the things that support us and allow us to move quickly.
1. Automation
The first pillar is automation. We try to automate everything we can but focus on two main areas: quality and security. First, we’ve automated testing, with more than 10,000 unit tests in the Android app and a suite of user interface tests. These tests run on every pull request and merge. Second, we do a release archive test in every pull request branch where we build the production version of the app that would get deployed to the store. Third, there are a lot of linting, quality, and security checks using tools like Detekt, Sonarqube, Black Duck, and Checkmarx. Last, we have data governance and compliance checks using an internal tool called Sysl.
We also use automation to remove repetitive processes, which increases our speed and reduces mistakes. For example, we auto-generate release notes and automate many of the change management artefacts that ANZ’s change management process needs.
2. No long lived branches
Our second pillar of continuous delivery is no long-lived branches. You may be familiar with git-flow, a popular strategy for branching and merging when using Git. We don’t use it. A “git-flow” style development process is inappropriate for continuous delivery.
Instead, we use a trunk-based development process where we have many short-lived branches off the main branch, all merging back into the main branch. Each branch should only be open for a few days and contain small manageable chunks of code that get merged via pull request.
3. Feature flagging
We make extensive use of feature flagging. New features are built and hidden behind flags, and we use a remote toggle solution to release features when they are ready. Hiding new code behind flags allows us to merge and ship without fear that we will expose incomplete features to users.
To allow us to test the application, we also have a companion app that the developers and testers install alongside the main ANZ Plus application. It allows us to override flag values to enable and test features under development.
Using flags decouples releasing features from deploying new builds into the app stores. We can move process-heavy items like legal approval and marketing away from when we deploy the app to when a feature is turned on and released. Developers don’t experience stress and pressure to finish something in time for arbitrary release dates because feature flags are not linked to the release cadence and can be turned “on and off” anytime.
4. Shift left
Our last pillar of continuous delivery is consciously “shifting left” as many of our quality assurance and security testing practices as possible. The earlier we find issues, the simpler they are to fix.
This means you’ll see a lot of activity in our pull requests before merging. You will first notice the automated testing and quality gates. Before merging a PR, all the automated unit and UI tests, linting, and security checks must pass. Every code change will cause checks to be re-run.
Second, developers also do a lot of manual and automated testing. Each pull request should come with unit & UI tests and a manual test plan authored by the developer, often with input from their quality coach. The developers reviewing the pull request’s code must also execute the test plan. Having developers be responsible for testing means they look at the code from different perspectives, including, most importantly, what it does when they run it.
Too often, when reviewing pull requests, I only focus on the style and architecture of the code, and it can be tough to see the forest for the trees when looking at a whole bunch of changes. It’s challenging to think about what the code does. Having a test plan in the pull request and having to execute it helps counteract this tendency and allows us to find problems early.
Detailed pull request reviews also require context. Pull requests usually have an extensive description, including screenshots and videos. All of this leads to much discussion on pull requests.
We don’t see this as a bad thing at all. Code is at the core of what we are doing, so it makes sense that an engaged and passionate team will spend a lot of time discussing and pulling it apart to improve it. Every time somebody opens a pull request or comments on one, the product improves, and we also grow and learn as a team.
Go slow to go fast
You might be thinking at this point, “Wow, I bet all this stuff takes heaps of time. All of that automation and all those extensive pull requests to review.” This brings us to a bit of a bonus pillar, or perhaps it’s more of an underlying principle: to go fast, you have to go slow.
“Eh? Go slow to go faster? Don’t you go fast to go fast? How does this make sense?”
The problem with trying to speed up by just “going faster” is that it doesn’t get you very far. If you’re under pressure to speed up, you can always cut some meetings and processes. However you quickly run out of these low-hanging fruit. There are limited number of things you can stop doing.
Then you start doing things like cutting corners on your code or putting off investments in automation. You start shortchanging your pull requests and get them approved quickly with a “looks good to me” comment. You think, “ah, it’s okay; I’ll fix it when the pressure is off.” Except somehow, the pressure never does come off. All those cut corners add up, and suddenly you find you’re going slow. Slower than you were before you tried to go fast!
So we don’t do this – instead, we deliberately “go slow to go fast.” It’s all about investing in improvement. We speed up every time somebody pauses to automate a process or spends time having detailed discussions in pull requests. These investments have a way of compounding over time. It is not an easy way of going fast, but it’s a powerful one that gives us a significant advantage.
