CI / CD Process Update: octopus deploy

imageThe third part of the article is about updating CI / CD processes. 

At this stage, we have (at least should have) an understanding of how CI will work, what we have at the input, how to get the result from it, and what needs to be done so that the build output becomes a full-fledged project module and gets started. As well as an established octopus (preferably LTS). In the last part, the process of setting up teamcity was considered: everything that happens from the moment when the code gets into the repository and until the ready archive can be unpacked, and in theory, it should work.

I remind the table of contents:

Part 1: what is, why it is not like, planning, a little bash. I would call this part near-technical.
Part 2: teamcity.
Part 3: octopus deploy.
Part 4: behind the scenes. Unpleasant moments, plans for the future, possibly FAQ. Most likely, it can also be called near-technical.

Octopus: General


At the input, we have the code, at the output, assembly artifacts, everything is clear here. But the result for each module consists of two parts: a core package (the result of the build itself), and an individual package (unique configurations and libraries). So, in order for a particular module to work, we need something like the following:

  1. An instance in IIS that has its own pool and that "looks" in document root. A separate pool is allocated for each module.
  2. Core package files in document root.
  3. Individual package files in document root.
  4. It is advisable to backup the old contents of document root in front of all this. We, of course, are confident in ourselves, but people are divided into two types: those who do not backup, and those who already do. 
  5. ( ) ( ). , , .
  6. , , .
  7. , .
  8. () .

Not that much, but not enough. There are no problems with step 1. Octopus has a Deploy to IIS template, it has everything you need. At steps 2 and 3 we will dwell in more detail below. For step 4, a prompted variable is sufficient, which we will pass from teamcity. Step 5 is a powershell script. He will also rotate backups. Why do we need to store them all, the old ones can be removed. Step 6 is also powershell. Step 7 - the finished Slack - Detailed Notification template. Step 8 - again powershell. It sounds like a plan, so we will implement it.

Tenants


A little more details about steps 2 and 3. I assume that the reader is familiar with octopus deploy. If not - there are wonderful (and only) articles right there on the hub They can give a basic understanding. Here they are: one and the second . So, we have tabs of projects, infrastructure, tenants, libraries, current tasks and settings. Necessary - tenants. The literal translation of the word tenant is tenant. In this context, most likely the customer, the client, was meant. Oktopus tells us that tenants allow you to deploy different instances of a project at once for many clients (tenants). This is the page with their list.

image

In this case, tenants are used a little differently. Each tenant represents an individual client. For the tenant there is the opportunity to add projects that will be launched when it is deployed. There will accordingly be a core project and an individual project.
There are two tags for each tenant: TenantRole - common for everyone, indicating that a module of a specific type (core) is deployed to all tenants with this tag, and ModName is a tag matching the module name (individual). 

image

The last octopus is received from timcity as a parameter, thus identifying the module that is currently deployed ( Deploy.Env and Deploy.2 ).

As you can see from the screen, when deploying a module for a specific organization, the core project will start first ( deploy.2 teamcity step), then an individual project ( steps deploy.3 and deploy.4 teamcity ). Each tenant has access to the variables of all projects that are included in it, as well as to the variables of the tenant itself (collected in a variable set, and contain the ModName tag).

image

 The tenant for the deployment is selected in the team as follows: 

  • env.tenantRoleTag variable - contains TenantRole (tenant tag name in octopus), and its actual value;
  • env.modName - The value of the ModName tag in octopus.

Variables


Variables are an important part of the project, and they need to be structured correctly. In this project, the octopus variables are divided into three types: common for all projects (collected in the Library set “Environment”), individual for each project (configured directly in the project), and variables directly from tenants (also a library set). In the Environment, as already mentioned, everything common for all projects, in particular: 

  • base paths to document root (for example, C: \ inetpub \);
  • base URL (for example, for all modules the common domain is organization.com);
  • access to databases (needed to organize backups);
  • etc.

A huge plus is that in the octopus for each environment there may be a variable value. The separation can be anything.

image

For example, lines 3, 6 and 7 on the screen. Line 3 is a common case. Everything that needs to be deployed to the production environment gets here. Lines 6 and 7 describe quite specific cases, for example: the production environment with the role “CompanyWebsite” (line 6), and the production environment with the tenant tag “orgznizationX”. This is necessary to identify the servers on which the package will get, because for organizationX, for example, a separate server is allocated.

The variables for the project include everything else. Something that is not common and differs for different environments, for example, the fully qualified domain name of the module. In general, it is compiled as # {Tenant.ModName}. # {BaseModuleDomain}, however, it may be different in some cases.

image

 Just such a case is presented in the picture.

Channels


Still worth talking about the channels. Briefly and quickly - channels allow you to set the deployment logic. Very straightforward, this is described in the official documentation, in particular here and here . For example, a release with only the stable tag can get into the production channel, etc. There are 4 channels in the project that coincide with the environments (development, test, staging, production). Releases fall into them according to pre-release tags. It is indicated in the build number format teamcity ( Build.General settings of the part about teamcity). From teamcity, the channel is transferred in step Deploy.2 to the additional command line arguments field (--channel parameter).

Patterns


If you have not yet appreciated all the advantages of using templates while reading an article about teamcity, now is the time. It is advisable to template custom steps that are common enough, because if there is a need to make edits, you will have to remember in which places this step is applied, click on each similar place and make edits as many times as the uses for this step. In this project, templates were created to check the site’s health and to send a list of domains back to teamcity. Briefly consider the process of creating a template.

There is a powershell script that needs to be run every time after the deployment of any module.

$url = "$scheme" + "://" + "$domain"
Write-Host "Requesting '$url'"
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
$request = [System.Net.WebRequest]::Create($url)
$request.Timeout = $timeout
$response = $request.GetResponse()
$response.Close()

The script accepts the input scheme, domain and timeout after which we believe that the site is not operational. You can create a template in Library - Step templates.

Enter a name, choose a logo. Next, configure the input parameters and their default values. The default value may also be the value of the specified variable (the value of the variable # {HostName} is the default value of the input parameter # {Domain}).

image

Next, we modify the script a little, and add it as a step to execution:

image

Done. At the output, we get a step template that is pleasant to work with, and also, in the absence of our work, colleagues will be able to figure out the field values ​​without reading the obscure scripts.

image

Octopus: process details


All projects are divided into groups, the description will be carried out on the example of the Modules group. It contains the following projects: 

  • core;
  • individual module projects.

Let's start with the core project.

Here is the overview page:

image

Here you can clearly see which versions of the kernel where they got. I apologize for the confusion in the version numbers, now we are switching to versioning tied to the git tag, and everywhere except production we already use the new versioning.

As you can see, for some tenants there are no defined environments. They are either not yet created or not foreseen. 

Deployment process core


At the stage of kernel delivery, part of the service operations are also performed, namely the backup of the database (if necessary), backup of the previous version of the files, rotation of backups and the kernel deployment. 

image

Step 1: backup the database. It is optional, and is based on the value of the variable parameter that is passed from teamcity in the Deploy step. 2 to the additional command line arguments field. By default, this variable is set to false, since the backup of the database should be done only on demand, and not constantly. In team city, starting with changing this parameter looks like this (you need to click on the ellipsis next to the run button):

image

In octopus, it looks like this: 

image

Next, nothing unusual. A backup copy of files in zip (by template), backup rotation (the script will be a bit lower), and a kernel deployment by template. 

What you should pay attention to: 

  • all significant parameters are set by variables, this allows you to quickly change their value in one place without digging deep into the process settings;
  • the IIS pool name matches the instance name and the service domain name (for convenience);
  • in the deploy IIS step, Merge with existing bindings is installed, instead of replacing. Since in addition to the service one, we still have a bunch of client domains at each instance;
  • Cleaning up document root (checkmark purge this directory before installation) fails. New files replace old ones as you copy. This is done so that the site does not “crash” during the deployment. The delay in loading the page is much nicer than the death screen.

Backup rotation script:

$days = $OctopusParameters["DaysStoreBackups"]
$limit = (Get-Date).AddDays(-$days)
$pathDb = $OctopusParameters["DbBackupDir"]
$pathFiles = $OctopusParameters["FilesBackupDir"] + '\' + $OctopusParameters["HostName"]

# Delete files older than the $limit.
Get-ChildItem -Path $pathDb -Recurse -Force | Where-Object { !$_.PSIsContainer -and $_.CreationTime -lt $limit } | Remove-Item -Force
# Delete any empty directories left behind after deleting the old files.
Get-ChildItem -Path $pathDb -Recurse -Force | Where-Object { $_.PSIsContainer -and (Get-ChildItem -Path $_.FullName -Recurse -Force | Where-Object { !$_.PSIsContainer }) -eq $null } | Remove-Item -Force -Recurse

# Delete files older than the $limit.
Get-ChildItem -Path $pathFiles -Recurse -Force | Where-Object { !$_.PSIsContainer -and $_.CreationTime -lt $limit } | Remove-Item -Force
# Delete any empty directories left behind after deleting the old files.
Get-ChildItem -Path $pathFiles -Recurse -Force | Where-Object { $_.PSIsContainer -and (Get-ChildItem -Path $_.FullName -Recurse -Force | Where-Object { !$_.PSIsContainer }) -eq $null } | Remove-Item -Force -Recurse

The deployment process of an individual package


At the stage of delivery of an individual module package, the configs and unique libraries are deployed, the files are located and / or renamed, the site is requested (firstly, sites on .net are loaded for the first time for a long time, this step allows you to perform this procedure immediately, and secondly it allows you to understand whether there are any errors when loading the site). A notification is also sent to slack, and an actual list of all domains belonging to this module is transferred to teamcity. 

image

From the important:

  • In the deploy step, document root is not cleaned up (the new core package is already there);

Step 3, the script. It was a little higher in the description of the templates.

Step 5. It is also associated with the deploy.4 step in teamcity.

In octopus, it looks like this:

$baseDomain = $OctopusParameters["HostName"]
$domains = (Get-WebBinding $baseDomain).bindingInformation | %{ $_.Split(':')[2] } | Sort-Object | Get-Unique
Write-Host "##teamcity[setParameter name='AllInstanceDomains' value='$domains']"
Write-Host "##teamcity[setParameter name='ServiceDomain' value='$baseDomain']"

The syntax ## teamcity ... allows you to set the build parameter in teamcity. Actually, in it, this step looks like this:

Write-Host "All domains on instance %ServiceDomain% :"
$domains = "%AllInstanceDomains%"
$domains = $($domains.Split(" ") | ForEach-Object {"http://$_"} | Out-String)
Write-Host $domains
Write-Host "Service domain: http://%ServiceDomain%"

Octopus: results


The whole build process is as follows:

  • code is collected from a specific branch (corresponding to the environment);
  • the result of the assembly (artifacts) is given to the octopus and consists of two parts: the kernel and an individual module package. This allows for efficient use of disk space and saves overall update delivery time. 
  • Each package has a version consisting of a number and a pre-release tag that defines the channel and the environment in which this release will fall.
  • Octopus, by input, creates a release and delivers packets to the appropriate servers, guided by the organization data (tenant tags), channels (pre-release tag) and additional instructions (optional backup of the database), and also makes an elementary check of the module’s performance.

That's kind of like that. Most likely, there will be another near-technical article, so to speak, behind the scenes. It will include everything that did not enter here: some details, what became new in the process of work and what you may have to face for those who decide on this, corrections and updates on the project, perhaps frequent questions / answers. Thank you for the attention.

All Articles