In the previous articles in this series, we discussed how to gather the needs and requirements of the business and then articulate them into measurable goals.
We also discussed how to staff the development team and which competencies/experiences to focus on and how they are linked to the goals. I also touched upon the agile principles and Scrum and how you can start using them in practice. In the subsequent article, I talked a bit about communication, collaboration, and the importance of getting to know each other. In this article, I will tell you about how you can implement the CI/CD mindset in your team and how it can ensure that your team's deliverables are always up to date.
Continuous integration (CI)
Getting started with CI requires some preparation. For example, setting up a repository where all developers can check in the code they are working on. Some popular products for code sharing are Github, Gitlab, and Bitbucket. Considering that Microsoft now owns Github, many may have concerns about entrusting all their source code to them. Fortunately, you can also choose to have a local installation where you don't hand over your code to an external company. Once you have decided how to handle your team's source code and how it is stored and version-controlled, it's time to start thinking about how you can automate the building and testing process. In a traditional development environment, developers create their code, compile it into executable binary files, and then it can be installed and launched in a test environment where it is tested both automatically and manually. The downside of this approach is that it becomes highly dependent on individuals and vulnerable to the risk of human error. People are not really designed to perform monotonous, unqualified tasks, but for a computer, it is the perfect day at work.
There is a plethora of tools to facilitate CI work, and many of them are tied to the source code management system you choose to use. Some popular tools worth mentioning include Jenkins, Argo CD, Tekton, CircleCI, GitLab CI, Github Actions, FluxCD, and more. Categorizing these tools isn't straightforward as their functionality overlaps with other areas, but if you explore these tools and search online, you will find numerous concrete examples of how to set them up and establish a smooth CI environment.
The next step is to ensure that the code is compiled and built automatically, which depends on the development environment you use and the programming or scripting languages you employ. In reality, there is no difference between manually compiling and building everything because the steps you need to define in the automated workflow are exactly the same. So, for example, if you build your Java-based applications with Maven, that is what you should define in the automated workflow.
Automated testing
Automating the tests we need to perform to deem the code mature enough and ready for deployment is a highly complex area. Most people are familiar with the classic test pyramid, with unit tests at the bottom, integration tests on the second level, and end-to-end tests at the top. However, what is not included as obvious parts of the test pyramid are performance testing, application security testing, and penetration testing. There are also many different aspects within each type of testing, making it even more complex. Fortunately, in this area as well, there are plenty of tools available for automating the testing process. For example, for scanning through static source code, you can use tools like SonarQube, Raxis, Kiuwan, Parasoft, Embold, Veracode, and many more. The list can be extensive.
Once the code has been tested, you can also scan your artifacts (images) using tools that can read them. Examples of tools in this domain include Jfrog Xray, Octopus, PyCharm, Datadog, and so on. Similarly, there are many tools you can use in this area, so it is best to form your own opinion and choose what suits you best. Developers themselves write unit tests, which are at a much lower level and test individual method calls, ensuring that the correct parameters are returned, for example. With the help of the scanning tools I mentioned earlier, it is possible to set up so-called toll gates that require a certain level of coverage of the unit tests that have been written. This way, you can, for instance, say that at least 75% of the code must be covered by unit tests before it can be approved and proceed. However, the problem with these strict boundaries is that it becomes easy for a developer to write tests just because they have to be done, not necessarily to ensure the code's quality. What I mean is that as a developer, I might submit my source code and receive feedback that it only has 73% coverage, while 75% is required. I can easily create some unit tests that invoke methods in my source code but may be entirely irrelevant for the application to function. Nevertheless, it contributes to reaching 76% coverage and passing the check.
Well, one can discuss automated testing for ages. The important thing to keep in mind is that you should always ensure that you test enough to guarantee the product's quality to the extent that you can confidently deploy it to production. The same applies to the other testing areas I mentioned earlier. The most challenging (almost impossible) area to test is exploratory testing, but to some extent, this may be facilitated by new modern AI tools. However, currently, there is not a sufficiently effective testing tool for that type of testing as far as I know.
Continuous deployment (CD)
Now we have reached the final step where we actually deploy our application in production, and everyone can start using it. Just like with automated testing, you need to carefully consider the steps required to deploy the application in production. Here, you can follow a similar approach and ensure that you first deploy to your staging environments. A common scenario is to have a development environment where all experimentation and new development takes place, then move to a system testing environment where others outside the team can also test the application. After that, you can deploy to an acceptance testing environment where you may involve certain external users to participate in testing. Once you know the steps that need to be executed, you can automate them in the CI tools you use.
What can be a bit tricky is when deployment involves not only new applications but also configuration changes that need to be made for the new version to be started up and used. But don't lose hope! Even this can be "easily" managed with various tools and methods. Simpler tools like Puppet, Ansible, Terraform, and Chef can be used to update configurations on multiple systems simultaneously and in a structured manner. More advanced tools include products like Kubernetes and Docker Swarm, where you also virtualize the different systems where your applications run, enabling the management of the "servers" as code. Lastly, you can also complement these tools with a graphical interface. For Kubernetes, for example, there's OpenShift from RedHat, which allows you to easily manage your server cluster consisting of small units called pods, where all applications run within a container-based ecosystem.
In this article series, I will try not to delve too deeply into various technical paradigm shifts. However, I think it's still important to have a basic understanding of different techniques and get a sense of what can and cannot be done with today's technology. Just like with everything else, it's important to work on continuous improvements and constantly explore how you can make your deployment process more efficient while ensuring greater quality.
But what happens when everything does not go smoothly in the team? In the next article, I will discuss the challenges of managing conflicts, stress, and other obstacles in the team.
