Improving our tech deployments – from 7 weeks to 10 minutes
Technology

Improving our tech deployments – from 7 weeks to 10 minutes

20th February 2020

When it comes to deploying new Delio technology, speed is everything. But how do you go faster without compromising on quality or burning out the team?

Working as quickly and efficiently as possible offers many benefits from both a technical and business point of view. For sales, this includes a quicker time to market. While for client support, it leads to implementing fixes to a client’s platform within SLAs and ensuring that the needs of the institutions using our tech are met. Everything we do at Delio is geared towards the customer – and our customers want new and improved features as quickly as possible into their product.

The Delio technology team recently began its journey to improve the sustainable pace of our work throughout the release cycle. Historically, our releases would take a long time to implement, with a 7 week lead time from an idea to production. However, by implementing automation best practices throughout the process, we managed to reduce our deployment time dramatically and have scope to reduce them even further. Deployments that took up to seven weeks are now done in ten minutes – improving our efficiency and causing significantly reduced downtime for our clients.

What our deployment process used to look like:

Once a new feature was chosen, and the user story created, multiple teams had to intervene throughout the process. But these teams would also have essential day-to-day tasks, which would take priority over releasing code – especially in terms of testing and staging environments.

This approach often caused a bottleneck in the release cycle. Releases often require a large amount of planning and are batched up and released on a single night. This process limits the deployment period, where new features and improvements can reach the production environment.

It was very clear that improvement was needed.

We quickly decided to invest in developing an automated deployment pipeline. One, with enough checks and balances to ensure that any code that is released is fully functional, secure and bug-free. While we are currently implementing this process up to the staging environment, our confidence in the automated pipeline has grown, and we will be looking to expand on the process even further.

So, here it is – our new automated process:

  1. Code is merged into the master git branch.

  2. CircleCI is triggered. This is where the unit, security and code quality tests are carried out.

  3. If all tests pass the code is built ready to be deployed in AWS.

  4. Depending on the pipeline, CircleCI will either upload static files to S3 or containers to AWS Elastic Container Registry (ECR).

  5. CircleCI sends a message to an AWS SQS queue. This message contains what pipeline is run, and all the required information Jenkins needs to deploy successfully.

  6. An AWS Lambda triggers Jenkins using the Jenkins API. The Lambda takes key information sent by the CircleCI SQS queue and uses it to determine what pipeline to run.

  7. Jenkins runs the following stages:

    • Deploy to a staging environment

    • Run feature tests in the staging environment

    • Deploy to a demo environment

    • Run feature tests in the demo environment

    • Deploy to production

    • Run feature tests in production

  8. At each stage of the pipeline results and events are reported back directly to the team’s Slack channel to support clear communication.

This new process dramatically reduces the amount of manual intervention required by external teams. The process consists of different ‘stages’. Once code is merged into the master branch, the pipeline ensures all code is tested. These tests include Unit Tests Security Tests and Code Standards to ensure the code is formatted correctly and meets the relevant criteria. By implementing a feedback loop through Slack, any issues during the deployment are raised instantly with the relevant development squad. We then hold the release until the team has investigated and resolved the issue.

So, at a high level, this is how the Delio technology team has adapted internal processes to improve our efficiency, speed and quality of work. In doing so, we have created a workflow that encourages smaller, more frequent releases and have improved the quality of our platforms in a shorter time frame. While this is not only rewarding from a technological perspective, most importantly, we are providing even greater value to our clients in the process. The next step is to roll out this new functionality across our full product range, so even more of our clients benefit from these advancements.