Continuous Integration and Delivery to Improve the Software Development Workflow

Are you looking for ways to improve your software development workflow? Then you are in luck as I may know a few guys. Most of them happen to work at Dreamix, our custom software development company. As we learned during jOnConf earlier this year, integrating and delivering software is an essential part of the software development process. So important, that the success of your business may rely on the quality of these particular processes.

This blog serves partly as complementary to my previous blog about what is a continuous integration and how to benefit from it, so if you are missing any pieces of the puzzle in your mind – go back and check it out. We will try to go a bit more deeply in the actual “how” of the processes and what you may want to add as part of their execution.

Now, let’s deep dive in.

Why do we Need Continuous Integration (CI)?

As we found out in what is a continuous integration and how to benefit from it, CI is a crucial part of the professional development of software. We want the newly introduced changes to be validated and tested through automated builds so that they don’t break existing functionality. We want to empower our developers to focus on what matters most and what brings them the most pleasure – writing code.

What we achieve through CI when appropriately done is:

• automating the tedious, repetitive manual labour
• enabling developers to focus on the development
• Allowing the developers to produce value faster
• accelerating delivery of new features
• frequent and short integration cycles / small Pull Requests (PRs)
• improving the quality of delivered features
• improving the quality of an existing codebase
• catching errors earlier, lowering the cost of errors and bugs
• preventing integration hell
• preventing code freeze due to changes causing major failures
• trust that what we produce is working
• saving time and effort
• producing software that gets better with time and is not deteriorating

Why do we Need Continuous Delivery (CD)?

As we already know from the 5 factors that lead to successful custom software development, CD (along with CI) is one of the tools that enable us to keep being successful in a sustainable way. CD continuously delivers the newly implemented functionalities to different user/target groups – developers, testers, product owners, end-users. It makes the functionalities available for use. CD is the next step in the software delivery process after CI. The delivery could be happening incrementally only for part of a specific group initially a.k.a. Canary Deployment before rolling out to all of the users.

We need continuous delivery because we want to move fast. We want to move with confidence and trust in what we deliver. We want what we deliver to be of high quality. And we want to transfer our investments and costs from repetitive, error-prone, and straightforward work to creative work so we gain competitive advantage and more value for our customers.

When Should we Implement and use CI and CD?

Here I am tempted to say “always”. But it will not always hold true. One of the most important factors in making this decision should come from the ratio – the cost to value. There are certain cases where CI and CD are not needed, not worth it or even not possible and cases where neglecting them or not using them will cost you much more in the long term than doing it the good old manual way.

We should implement and use CI and CD when we:

• are dealing with a high frequency of integration and delivery for our software
• have an automatic test suite that verifies the integrity of our software
• are dealing with large-scale enterprise applications with high availability and durability requirements with a lot of software developers working on the code
• want to have bigger transparency within the dev team about the state of the source code, the software, and environments
• want to be sustainably delivering measurable, specific, and high-quality results.
• want to increase dev team productivity.
• want to transfer knowledge explicitly so it is available and active
• ly used by anybody in the future – though tests that are executed on the codebase as part of the CI/CD processes
• want to enable each member of the team to focus on what he is good at and the activities in which he delivers the greatest value with the highest quality possible that brings him the greatest pleasure
• have a very high cost of bugs that are found on production, e.g. our reputation or revenue becomes at stake critically
• will have impediments on the functioning of our every business if a bug is present on production and fixing a bug on production becomes the top priority of the whole team regularly

We may NOT want to implement and use CI and CD when we:

• are not integrating and delivering often
• are working in a team consisting of one developer
• are not developing and maintaining automation tests
• have little functionality available that needs testing and verification
• are creating small-scale applications that are available for a very limited group of users

How To and Best Practices

For the best results, we need the proper tools. There are a few factors that we are interested in when choosing our direction and choice of tools. One question we would like to answer ourselves would be “how big is the community behind this tool/platform?”.
Apart from the tools, you will need a team from experienced experts that will be able to guide you through the process.

Let’s not forget that actually, it is the content that contains the value. And in this context, this means that we need fitting processes that will fulfill our needs. We need to consider what elements do we want to include as part of our CI/CD processes a.k.a. pipelines.

Example steps/elements (not an exhaustive list) of a CI pipeline for a Maven project:

Verify any global configuration that is needed is in place, e.g. credentials

1. Check out the source code repo from git
2. Ensure we are on the main/desired branch
3. Set the new version of the Maven project
4. Build the project, along with all tests that are testing the integrity of the software
5. Build any necessary docker images/helm charts, etc.
6. Upload any generated artefacts to the desired repositories – jars/wars in nexus, docker images to the docker repositories, helm charts to the ChartMuseum, etc.
7. Create a preview environment from the generated artefacts by deploying them there
8. At this point, you may even want to execute some integration tests that verifies the interaction between the preview environment and other environments / external systems
9. Post the results from the pipeline to the actual PR that triggered it / send emails with results
   Clear workspace where we were executing this pipeline

Example steps/elements (not an exhaustive list) of a CD pipeline for a Maven project:

1. Define the version number of the release, prepare the necessary configuration
2. Check out the source code repo from git + checkout the specific environment configuration repo from git
3. Ensure we are on the main/desired branches
4. Set the release version to the release artefacts that will be deployed
5. Generate the release artefacts
6. Upload any generated artefacts to the desired repositories
7. Run the maven command for deploying the project
8. Generate changelog for the env, publish it to the git environment configuration repo
9. Promote/deploy the release artefacts to the desired environment
10. Clear workspace where we were executing this pipelin

Note: “git environment configuration repo” refers to a repo in git that is containing all of your configurations for the environment where you want to deploy. You can find examples of Jenkins X staging environment repos here

Make sure everybody in the team knows and understands what is the purpose and the value of each part of all processes and why the team is doing something.

For more and advanced information on the topic I strongly recommend you read the following books:

• https://www.amazon.com/Accelerate-Software-Performing-Technology-Organizations/dp/1942788339
• https://github.com/vfarcic/devops26

2 of my personal favorite tools for CI and CD are Jenkins and Jenkins X

Bonus

Ten Commandments Of GitOps Applied To Continuous Delivery

• Git is the only source of truth.
• Everything must be tracked, every action must be reproducible, and everything must be idempotent.
• Communication between processes must be asynchronous.
• Processes should run for as long as needed, but not longer.
• All binaries must be stored in registries.
• Information about all the releases must be stored in environment-specific repositories or branches.
• Everything must follow the same coding practices.
• All deployments must be idempotent.
• Git webhooks are the only ones allowed to initiate a change that will be applied to the system.
• All the tools must be able to speak with each other through APIs

Now What?

Looking for a short consultation to find out if CI and CD are for you? We can lend you a helping hand at only a single message distance.