Deploy only what changed with Octopus

Big Red Deploy Button

Jason Reardon

04/16/2018

Imagine this, if you will... you have started working on a new project for your company with a new solution and a new set of services. Maybe you have all the services framed out, but with minimal code or maybe you're just starting development on a few services. Your company is doing things right: a build server immediately picks up commits as they're pushed and kicks off a continuous integration build. Once the build is complete and all the unit tests pass, your build server hands the packages off to an Octopus Deploy server to be deployed. Maybe you even have the deployment automatically trigger and within minutes of pushing your latest changes, they're live in the development environment. While this is great in the beginning, as your codebase grows and your services are filled out, those deployments start to take longer and longer. What was mere minutes in the beginning, starts to take 15 minutes, then 20, and maybe even longer. If you have an active team regularly pushing out changes, builds might even back up, and it could take upwards of an hour for your code to be deployed and ready for testing. This is not an optimal situation. But what can you do? It takes time to copy all the service packages to your Octopus server, to stop all the services, then deploy the new ones, and start them all back up again. So, you wonder, "what if I could deploy only the services that actually changed?" Well, wonder no more, because I have a solution for you.

Before We Begin

For this demonstration, I am going to be using TeamCity with Octopus Deploy. While this process is specific to Octopus, any build agent that can run scripts will work in place of TeamCity. Also, this is not exactly a step-by-step guide on how to achieve this. You will need to be familiar with how to set up builds in your preferred build platform and how to configure the deployment process in Octopus. The main purpose of this article is to convey the techniques and settings used to achieve the desired outcome.

Before we even get to the build, we'll need to do some prep work within the solution itself. Since we're using Octopus, we'll need to install the OctoPack NuGet package into all the service projects which will be deployed. OctoPack is nifty a tool that packages up all the necessary files from a project's bin directory into a nupkg file after it's been built. It looks for a nuspec file to tell it how to package up the files but will create a default nuspec if one is not provided. In our example, we will be adding a nuspec to each project, so we can tell which projects need to be deployed, this could also be done in an automated fashion, which we'll discuss later. You can read more about OctoPack here.


Package Creation

Now we'll move on to the build process. First, make sure your build platform is setup for Octopus. Some will come with Octopus functionality built in, but others like TeamCity will require that you install a plugin. You will want to set up a build as you normally would. Once the plugin is installed into TeamCity, the Build step template will have an Octopus Packaging section. All you need to do here is check the Run OctoPack box and enter a package version number. I like to use the Build Number for this; just make sure that it will be a different number with every build, as Octopus will only accept packages with unique names. 

Next, we will add a PowerShell step (this could be whatever scripting language you feel most comfortable with). This is where things can be a little different based on which build platform you're using, but the process will be roughly the same. TeamCity supplies us with a convenient changed files file, which is accessible with the system variable system.teamcity.build.changedFiles.file. This can be parsed for changes to specific project directories or files. If the build platform you are using does not have a file like this, there is likely an API available, where you can pull down a previous build and compare the files to your current build. Or you might be able to check the commits in your build for which files have changed. In our case, we will be using the TeamCity changed files file and check it for references to any nuspec files. Since our projects are packaged up in the format of [PROJECT NAME].[BUILD NUMBER].nupkg and the id value in the nuspec file is set to the project name, we can iterate over each changed nuspec file and create a list of packages that need to be deployed using this information.

PowerShell script to parse the changed files file for a list of packages to deploy

Now that we know which projects have changes and need to be deployed, we can set up the deployment process in TeamCity. While all of this can go into one configuration, I prefer to separate the build and package configuration from the deployment configuration. This way the build and package configuration can be triggered on every branch (think feature branches) and the deployment configuration can be triggered only for branches/changes which we wish to deploy. If you decide to use a separate deploy configuration, make sure to add your packaged project files and the changed projects file to the package and build configuration's artifacts. 

Publishing Packages to Octopus

The deployment process will consist of three steps: staging the necessary packages for deployment, pushing the packages to the Octopus server, and creating and potentially triggering a release. In the first step, we add another PowerShell script, which will loop through the file which we created with our updated packages, then copy each package to a separate "DeploymentPackages" directory. 

PowerShell script to stage the packages to be published to Octopus

The second step is a built-in "OctopusDeploy: Push packages" step templated, which we tell to push every nupkg file that is in our DeploymentPackages directory. 

Package paths setting in TeamCity, these packages will be published to Octopus

Finally, in the third step, we use the octo.exe application to create a release with a channel specific to each of our changed projects (which will be explained further in a moment). While we are simply creating a release for manual deployment, the release can be automatically triggered with octo.exe.

PowerShell script to create and optionally deploy a release for a specified channel

Configuring the Deployment Process in Octopus

Now we move on to setting up Octopus Deploy, where the magic really happens for deploying services individually. While the main feature here is the Channel, we will start by setting up a deployment step for each deploy-able service inside of our solution. Once that is finished, we set up one Channel for each of the services that we created a deployment step for; which, for our automation purposes needs to be named after the project (this should match the 'id' value in the nuspec file). Inside of the Channel, we need to create a Version Rule. For the rule, under Package step(s), we add the deployment step(s) that correspond to the package which the channel was created for. A Version range is required, so we can enter an asterisk '*' to signify 'all versions'. Once this is finished, we need to go back into our deployment steps and set the Conditions on each one to only run for the Channel for the corresponding package. Once the Channels and deployment steps are set up, this should look something the following:

Version Rule example in Octopus
Step conditions example in Octopus


And that's all there is to it. The version rule tells Octopus that it only needs to create a release for packages required by the defined steps, that fit the specified version range. Then, adding the channel to the deployment steps tells those steps to only execute when the release is being run inside of that channel. This way, we can limit each service to be individually deployed. And, if there are other steps that you want to run on every release, just leave the Channel condition out.

Thank you for taking the time to read over this article. If you have any questions, comments, or concerns, please sound off in the comment section below. I will try my best to answer any questions or address any concerns.