How to start testing Ansible, refactor a project for a year and not fly off the coils

This is the transcript of the performance on DevOps-40 2020-03-18 :

Starting from the second commit, any code becomes legacy, because initial ideas begin to diverge from harsh reality. This is neither good nor bad, it is a given with which it is difficult to argue and it is necessary to get along. Part of this process is refactoring. Refactoring Infrastructure as Code. Let the story begin how to refactor Ansible in a year and not fly off the coils.

The Legacy Origin

Day 1: Zero Patient

Once upon a time there was a conditional project. It had a Dev development team and Ops engineers. They solved the same problem: how to deploy the server and run the application. The problem was that each team solved this problem in its own way. The project decided to use Ansible to synchronize knowledge between the Dev and Ops teams.

Day No. 89: The Birth of Legacy

, , legacy. ?

• , — .
• .
• Ansible / Python / Bash / Terraform! !
• Full Stack Overflow Developer stackoverflow, , .

, , , , , , , , .

- hosts: localhost
  tasks:
    - shell: echo -n Z >> a.txt && cat a.txt
      register: output
      delay: 1
      retries: 5
      until: not output.stdout.find("ZZZ")

№ 109:

IaC / / , . , .

IaC

№ 139: ?

:

1. ?
2. ?
3. ?

, . .. ( , 200 000 ), .

№ 149:

e . . , . - , confluence, " ?" " ?" . : / . , , - .

, . , , : , , .

Ansible

Ansible , , .

№ -997: SDS provision

Ansible SDS (Software Defined Storage).
, , 60-90 , . e2e , .. , . . .

№ -701: Ansible test kitchen

Ansible , test kitchen / kitchen-ci inspec. Ruby ( : YML ansible?) 40 10 . .

, - . 13 2 , 70 , 2 . XP (extreme programming) .. 70 .

№ -601: Ansible molecule

testkitchen, docker . , 20-25 7 .

17 45 28 2 jenkins slave.

№ 167: Ansible

, . , . , .

, , , jira, google docs . , , . , - , , .

:

• Eat.
• Sleep.
• Code.
• IaC test.
• Repeat

.

, .

№ 181: Green Build Master

Green Build Master. , jenkins. , :

• .
• - , .

№ 193: unit

— , . , .

№ 211: unit integration

unit , . .. , , .

jenkins , / , .

Jenkins + Docker + Ansible = Tests

1. Checkout repo and generate build stages.
2. Run lint playbook stages in parallel.
3. Run lint role stages in parallel.
4. Run syntax check role stages in parallel.
5. Run test role stages in parallel.
   
   1. Lint role.
   2. Check dependency on other roles.
   3. Check syntax.
   4. Create docker instance
   5. Run molecule/default/playbook.yml.
   6. Check idempotency.
6. Run integration tests
7. Finish

№ 271: Bus Factor

- . e. code review . , , , .. .

. , , . jenkins + bitbucket + jira.

, -, , :

- get_url:
    url: "{{ actk_certs }}/{{ item.1 }}"
    dest: "{{ actk_src_tmp }}/"
    username: "{{ actk_mvn_user }}"
    password: "{{ actk_mvn_pass }}"
  with_subelements:
    - "{{ actk_cert_list }}"
    - "{{ actk_certs }}"
  delegate_to: localhost

- copy:
    src: "{{ actk_src_tmp }}/{{ item.1 }}"
    dest: "{{ actk_dst_tmp }}"
  with_subelements:
    - "{{ actk_cert_list }}"
    - "{{ actk_certs }}"

, .

get_url:
    url: "{{ actk_certs }}/{{ actk_item }}"
    dest: "{{ actk_src_tmp }}/{{ actk_item }}"
    username: "{{ actk_mvn_user }}"
    password: "{{ actk_mvn_pass }}"
  loop_control:
    loop_var: actk_item
  with_items: "{{ actk_cert_list }}"
  delegate_to: localhost

- copy:
    src: "{{ actk_src_tmp }}/{{ actk_item }}"
    dest: "{{ actk_dst_tmp }}"
  loop_control:
    loop_var: actk_item
  with_items: "{{ actk_cert_list }}"

№ 311:

, . " , ". docker, . testinfra ansible verifier - .

:

1. docker.
2. , .
3. - .
4. .
5. .

Pipeline jenkins

1. Generate build stages.
2. Lint all in parallel.
3. Run test role stages in parallel.
4. Finish.

Lessons learned

Avoid global variables

Ansible , workaround private_role_vars, .

. role_a role_b

# cat role_a/defaults/main.yml
---
msg: a

# cat role_a/tasks/main.yml
---
- debug:
    msg: role_a={{ msg }}

# cat role_b/defaults/main.yml
---
msg: b

# cat role_b/tasks/main.yml
---
- set_fact:
    msg: b
- debug:
    msg: role_b={{ msg }}

- hosts: localhost
  vars:
    msg: hello
  roles:
    - role: role_a
    - role: role_b
  tasks:
    - debug:
        msg: play={{msg}}

, , . Ansible , - , .

BAD: .

# cat roles/some_role/tasks/main.yml
---
debug:
  var: java_home

GOOD: defaults .

# cat roles/some_role/defaults/main.yml
---
r__java_home:
 "{{ java_home | default('/path') }}"

# cat roles/some_role/tasks/main.yml
---
debug:
  var: r__java_home

Prefix role variables

BAD: .

# cat roles/some_role/defaults/main.yml
---
db_port: 5432

GOOD: , inventory .

# cat roles/some_role/defaults/main.yml
---
some_role__db_port: 5432

Use loop control variable

BAD: item, / -

---
- hosts: localhost
  tasks:
    - debug:
        msg: "{{ item }}"
      loop:
        - item1
        - item2

GOOD: loop_var.

---
- hosts: localhost
  tasks:
    - debug:
        msg: "{{ item_name }}"
      loop:
        - item1
        - item2
      loop_control:
        loop_var: item_name

Check input variables

, , ,

GOOD: .

- name: "Verify that required string variables are defined"
  assert:
    that: ahs_var is defined and ahs_var | length > 0 and ahs_var != None
    fail_msg: "{{ ahs_var }} needs to be set for the role to work "
    success_msg: "Required variables {{ ahs_var }} is defined"
  loop_control:
    loop_var: ahs_var
  with_items:
    - ahs_item1
    - ahs_item2
    - ahs_item3

Avoid hashes dictionaries, use flat structure

hash/dictionary , , hash/dictionary, .

BAD: hash/dictionary.

---
user:
  name: admin
  group: admin

GOOD: .

---
user_name: admin
user_group: "{{ user_name }}"

Create idempotent playbooks & roles

, .. configuration drift -. molecule, .

Avoid using command shell modules

shell , , Ansible.

Test your roles via molecule

Molecule , .

Molecule Multiple instances

molecule.yml platforms .

---
    driver:
      name: docker
    platforms:
      - name: postgresql-instance
        hostname: postgresql-instance
        image: registry.example.com/postgres10:latest
        pre_build_image: true
        override_command: false
        network_mode: host
      - name: app-instance
        hostname: app-instance
        pre_build_image: true
        image: registry.example.com/docker_centos_ansible_tests
        network_mode: host

, converge.yml :

---
- name: Converge all
  hosts: all
  vars:
    ansible_user: root
  roles:
    - role: some_role

- name: Converge db
  hosts: db-instance
  roles:
    - role: some_db_role

- name: Converge app
  hosts: app-instance
  roles:
    - role: some_app_role

Ansible verifier

molecule ansible , , 3 . testinfra/inspec, , :

---
- name: Verify
  hosts: all
  tasks:
    - name: copy config
      copy:
        src: expected_standalone.conf
        dest: /root/wildfly/bin/standalone.conf
        mode: "0644"
        owner: root
        group: root
      register: config_copy_result

    - name: Certify that standalone.conf changed
      assert:
        that: not config_copy_result.changed

, smoke test:

---
  - name: Verify
    hosts: solr
    tasks:
      - command: /blah/solr/bin/solr start -s /solr_home -p 8983 -force
      - uri:
          url: http://127.0.0.1:8983/solr
          method: GET
          status_code: 200
        register: uri_result
        until: uri_result is not failed
        retries: 12
        delay: 10
      - name: Post documents to solr
        command: /blah/solr/bin/post -c master /exampledocs/books.csv

Put complex logic into modules & plugins

Ansible , , , shell , . , , .

Summarize Tips & Tricks

1. Avoid global variables.
2. Prefix role variables.
3. Use loop control variable.
4. Check input variables.
5. Avoid hashes dictionaries, use flat structure.
6. Create idempotent playbooks & roles.
7. Avoid using command shell modules.
8. Test your roles via molecule.
9. Put complex logic into modules & plugins.

, IaC. , .

Links

• Slides How to test Ansible and don't go nuts
• Video How to test Ansible and don't go nuts
• , 200 000
• Ansible: 120 VM c Coreos Centos 18
• YML ansible?
• A list of awesome IaC testing articles, speeches & links
• English version

UPD1 2020.05.01 20:30 — callback_whitelist = profile_tasks . ansible. mitogen
UPD2 2020.05.03 16:34 — English version