Get best of the week

Deploy Jenkins as code

Note perev .: this is a translation of an article from Preply 's engineering blog about how you can use configuration as code for such a popular CI / CD tool like Jenkins.

In our company, we try to follow the practice of "Everything is like code", this applies not only to infrastructure resources, but also to monitoring, Jenkins work, etc. In this article, I’ll talk about how we use this practice to deploy and support Jenkins. And this applies not only to the infrastructure for the server and agents, but also to plug-ins, accesses, work, and many other things.

In addition, in this article we will try to find answers to questions such as:

  • Has our Jenkins become more stable?
  • Can we make frequent server and job configuration changes?
  • Is the Jenkins update still a pain for us?
  • Can we control all our changes?
  • Can we quickly restore Jenkins in case of a fakap?

image


Introduction


Usually, the first thing that comes to mind when mentioning the phrase “DevOps tools” is the CI / CD system. For example, we use Jenkins because we run hundreds of tasks every day, and that’s tens of thousands of builds. Some features that we use in Jenkins are either not available in other CI / CD systems or have limited functionality.


We would like to control Jenkins completely from the code, including infrastructure, configurations, job and plug-ins. We tried to run Jenkins in Kubernetes, but it did not fit our needs, plus it was not easy to scale due to its architecture .

image
This will be discussed

Infrastructure for Jenkins


imageWe use AWS and configure the entire infrastructure using Terraform and other hash tools such as Packer and Vault .

As mentioned earlier, we tried using Jenkins in Kubernetes and faced some problems with scaling PVC , resources, and a not-so-well-designed architecture.

Here, we use the usual AWS resources: EC2 instances, SSL certificates, balancers, Cloudfront, etc. The OS image ( AMI ) is configured using Packer, which integrates perfectly with Terraform and Vault.

{
    "variables": {
        "aws_access_key": "{{vault `packer/aws_access_key_id` `key`}}",
        "aws_secret_key": "{{vault `packer/aws_secret_access_key` `key`}}",
        "aws_region": "{{vault `packer/aws_region` `key`}}",
        "vault_token": "{{env `VAULT_TOKEN`}}"
    },
    "builders": [{
        "access_key": "{{ user `aws_access_key` }}",
        "secret_key": "{{ user `aws_secret_key` }}",
        "region": "{{ user `aws_region` }}",
        "type": "amazon-ebs",
        "communicator": "ssh",
        "ssh_username": "ubuntu",
        "instance_type": "c5.xlarge",
        "security_group_id": "sg-12345",
        "iam_instance_profile": "packer-role-profile",
        "ami_name": "packer-jenkins-master-{{timestamp}}",
        "ami_description": "Jenkins master image",
        "launch_block_device_mappings": [{
            "device_name": "/dev/sda1",
            "volume_size": 50,
            "volume_type": "gp2",
            "delete_on_termination": true
        }],
        "source_ami_filter": {
            "filters": {
                "virtualization-type": "hvm",
                "name": "ubuntu/images/*ubuntu-bionic-18.04-amd64-server-*",
                "root-device-type": "ebs"
            },
            "owners": ["099720109477"],
            "most_recent": true
        }
    }],
    "provisioners": [{
        "type": "shell",
        "environment_vars": ["VAULT_TOKEN={{ user `vault_token` }}"],
        "scripts": ["packer_bootstrap.sh"]
    }]
}
An example of what the configuration of an OS image looks like in Packer.

In turn, the file packer_bootstrap.shcontains a set of commands with which software is installed inside the image. For example, we can install Docker, docker-compose and vaultenv or Datadog-agent for monitoring. Regarding the infrastructure for this image, we can use Terraform, Cloudformation, Pulumi or even Ansible.

Here is an example of a possible AWS infrastructure for Jenkins.

Users log in to Jenkins through an internal balancer, and Github hooks get to a server through an external one. We use Jenkins integration with GitHub, so some server links must be accessible from the Internet. There are many different solutions here (for example, a white list for IP addresses , URLs or tokens, etc.), in our case we use a combination of allowed URLs and token validation.
So, after the manipulations we have done, we already have a ready-made infrastructure with the assembled OS image, the ability to monitor and access to the corporate secret store.

We use Docker to install Jenkins and its plugins


imageThe next thing we'll do is install Jenkins and its plugins. We constantly had problems updating the plugins, so the main goal was to have a clear cast of the installed plugins and their versions in the code.
And here Docker will help us, because we can take a ready-made pre - installed Docker image and use it as a base for our configuration.

FROM jenkins/jenkins:2.215

ENV CASC_JENKINS_CONFIG /jenkins_configs
USER root

#   
RUN apt update && \
    apt install -y python3 python3-pip && \
    pip3 install awscli jenkins-job-builder jjb-reactive-choice-param --no-cache-dir

USER jenkins
VOLUME /jenkins_configs
VOLUME /var/jenkins_home

#  
COPY plugins.txt /usr/share/jenkins/ref/
RUN /usr/local/bin/install-plugins.sh < /usr/share/jenkins/ref/plugins.txt
Dockerfile

Inside the Docker image, some packages are installed like Job Builder, which I will discuss later, the repositories are also registered and the plugins specified in the file are installed plugins.txt.

Jenkins.instance.pluginManager.plugins.each{
  plugin ->
    println ("${plugin.getShortName()}:${plugin.getVersion()}")
}
You can get a list of installed plugins in Jenkins by clicking on the link https://our-jenkins-url/scriptand saving the output to a fileplugins.txt

. Finally, the configuration for docker-compose, which will run Jenkins in Docker.

version: "3"
services:
  jenkins:
    build: .
    container_name: jenkins
    restart: always
    ports:
      - "50000:50000"
      - "8080:8080"
    volumes:
      - ./configs/:/jenkins_configs/:ro
      - ./jenkins_home/:/var/jenkins_home/:rw
    environment:
      - VAULT_TOKEN
      - GITHUB_TOKEN
      - AWS_ACCESS_KEY_ID
      - AWS_SECRET_ACCESS_KEY
      - JAVA_OPTS=-Xms4G -Xmx8G -Xloggc:/var/jenkins_home/gc-%t.log -XX:NumberOfGCLogFiles=5 -XX:+UseGCLogFileRotation -XX:GCLogFileSize=20m -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintHeapAtGC -XX:+PrintGCCause -XX:+PrintTenuringDistribution -XX:+PrintReferenceGC -XX:+PrintAdaptiveSizePolicy -XX:+UseG1GC -XX:+ExplicitGCInvokesConcurrent -XX:+ParallelRefProcEnabled -XX:+UseStringDeduplication -XX:+UnlockExperimentalVMOptions -XX:G1NewSizePercent=20 -XX:+UnlockDiagnosticVMOptions -XX:G1SummarizeRSetStatsPeriod=1
volumes:
  configs:
    driver: local
  jenkins_home:
    driver: local
We also use vaultenv to forge secrets from Vault.

Note some Java options that helped us with garbage collection and resource limits. This article is very cool about tuning Jenkins.

And of course, now we can locally deploy a copy of Jenkins and experiment with new versions of the server and plugins. It is very comfortable.
Now we have a clean installation of Jenkins and plugins, which can easily be launched in the prod. Let's add more configuration for her.

Configuring the Jenkins as a Code (JCaSC) plugin for server configuration


imageIn general, there is a plugin called Jenkins Configuration as Code ( JCasC ), which allows you to store the server configuration in human-readable text format.

Using this plugin, you can describe security configurations, accesses, settings for plugins, agents, tabs, and much more.

The configuration is presented in YAML format and is divided into 5 blocks:

  • credentials (description of system secrets)
  • jenkins (authorization and cloud settings, global settings, agent descriptions, some security settings and tabs)
  • security (global security settings, such as allowed scripts)
  • tool (configuration for external tools like git, allure, etc.)
  • unclassified (other settings, such as integration with Slack)

imageimageThe plugin supports importing configurations from an existing Jenkins installation.

In addition, the plugin provides support for various secret providers , however, in this example we will just use environment variables.

credentials:
  system:
    domainCredentials:
    - credentials:
      - usernamePassword:
          description: "AWS credentials"
          id: "aws-creds"
          password: ${AWS_SECRET_ACCESS_KEY}
          scope: GLOBAL
          username: ${AWS_ACCESS_KEY_ID}
      - string:
          description: "Vault token"
          id: "vault-token"
          scope: GLOBAL
          secret: ${VAULT_TOKEN}
      ...
This is how secrets can be described.

We also use the Amazon EC2 plugin to raise agents in AWS, and its configuration can also be described using this plugin. Matrix authorization allows us to configure user access using code.

jenkins:
  authorizationStrategy:
    projectMatrix:
      permissions:
      - "Overall/Administer:ivan.a@example.org"
      - "Credentials/View:petr.d@example.org"
      ...
  clouds:
  - amazonEC2:
      cloudName: "AWS"
      privateKey: ${EC2_PRIVATE_KEY}
      region: "${AWS_REGION}"
      templates:
      - ami: "ami-12345678"
        amiType:
          unixData:
            sshPort: "22"
        connectionStrategy: PRIVATE_IP
        deleteRootOnTermination: true
        description: "jenkins_agent"
        idleTerminationMinutes: "20"
        instanceCapStr: "100"
        minimumNumberOfInstances: 0
        mode: EXCLUSIVE
        numExecutors: 1
        remoteAdmin: "jenkins"
        remoteFS: "/home/jenkins"
        securityGroups: "sg-12345678"
        subnetId: "subnet-12345678"
        type: C52xlarge
        ...
Description of agents and accesses

The plugin supports some other things that we use. With a properly organized Jenkins local testing process, you can effectively find and fix bugs before they can potentially get into Jenkins sales.
Now we have a reproducible configuration for the server, it remains the case for small, namely - the job.

We use Job Builder for freestyle projects


imageThere are several ways to create freestyle jobs in Jenkins:

  • using the web interface (the easiest way, jumped on and went on)
  • directly using the REST API
  • using plugins like Job DSL or JJB wrapper

Jenkins Job Builder (JJB) allows you to configure jobs using YAML or JSON. And it’s quite convenient, because you can configure all the jobs and store their state in conditional git. That is, in fact, we can build a CI / CD process for our CI / CD tool using JJB.

.
├── config.ini
├── jobs
│   ├── Job1.yaml
│   | ...
│   └── Job2.yaml
└── scripts
    ├── job1.sh
    | ...
    └── job2.sh
This is how (simplified) the structure of the configuration of the job on the FS looks

- job:
    name: Job1
    project-type: freestyle
    auth-token: mytoken
    disabled: false
    concurrent: false
    node: jenkins_agent
    triggers:
      - timed: '0 3 * * *'
    builders:
      - shell:
          !include-raw: ../scripts/job1.sh
And this is how the job in the file looks Job1.yaml, steps in the script. job1.sh

The JJB configuration file also looks simple.

$ cat config.ini
[job_builder]
ignore_cache=True
exclude=jobs/Job2

[jenkins]
url=https://jenkins.example.org
user=some_user
password=some_password

$ jenkins-jobs --conf config.ini test -r jobs/
$ jenkins-jobs --conf config.ini update -r jobs/
The application of new changes can easily be launched using the Byjenkins-jobs update

itself command , the user for whom the token was created must have the appropriate privileges to create and configure the job. We just need to apply one initialization job (seed job), which will apply the changes using JJB in Jenkins.

It is worth mentioning that JJB is not a “silver bullet”, as some not very popular plugins are not supported. However, it is a very flexible tool for storing job in code, including macro support .

Summary


Now that we have reached the end of this article, I would like to return to the beginning and answer the questions asked at the beginning. We can answer “yes” to each of the questions posed.

In this article, we did not delve into the subtleties of setting up certain technologies or how to configure Jenkins correctly , we just share our experience, which may be useful to you as well.

PS In the article I often use the word “job” (from the English “job”, “task”), for me it sounds more familiar than “task” in the context of CI / CD in general or Jenkins.

More articles:


All Articles