GitLab CI/CD: test automation and application deployment

For a long time people have been asking me to prepare a tutorial on how to use CI/CD on GitLab. So I've put together a basic sample project and here's a tutorial on GitLab and its CI/CD.

How to get CI/CD working in GitLab

This tutorial lists the steps you need to complete to get your software project's automatic development and deployment up and running. This demo is focused on GitLab, but you can implement similarly on other solutions such as GitHub action.

What you need to have, provide and set up:

• Use GitLab.
• Have runnners.
• Set up a development process so that containers are used. For example, Docker.
• Split applications so that each service runs in a separate container.
• Have tests. Then set up automatic startup.
• Have a VPS to run the application.
• Setup nginx on the VPS. Create a user and set up ssh access to the server.
• Design an image build process for production and set up image build automation on gitlab.
• Set permissions for deploy.
• Execute build and push image to registry.
• Run deploy on gitlab - action will run on merge to master/main branch.

Deadlines

To use something you need to understand it. That's why with any new technology you need to explain the terms.

What is a runner?

A runner is an instance of GitLab Runner that allows you to run different types of tasks such as building, testing, and deploying a project. GitLab Runner is part of GitLab and allows you to automate common software development operations.

If you use gitlab.com by default you have runners available. If you are running GitLab on your own infrastructure, you will need to provide Gitlab runners yourself. That is, provision the servers, install the runner, and configure the connection to GitLab.

Details how to install the GitLab runner.

The runner then launches containers, according to the image you choose, and executes the specific job - the job you want to execute.

Pipeline, Job

A Pipeline is a set of steps that are executed to achieve a specific goal, such as deploying an application to production. A pipeline usually contains several phases (stages) that include different steps such as build, test, and deploy. Each stage may contain one or more jobs. Pipelines can be triggered automatically whenever there is a change in the code or based on a specific schedule.

A Job is a specific task that is executed within a pipeline. Each job is a unit of work that can be independently run and monitored. Jobs can be divided into different phases of the pipeline. Each job may contain scripts or commands that need to be executed, such as running tests, building code, or deploying an application.

A pipeline consists of one or more stages. A stage can consist of one or more jobs.
Stage is run in sequence. Multiple jobs in a stage are run parallel.

What are docker container registries?

Docker Container Registry (also known as Docker Registry) is a service for storing and managing Docker images. It is a central repository where you can store your own Docker images and share them with other people in your organization or publicly on the Internet.

GitLab includes Container registries. When you do a build on GitLab, it's not an easy solution to use GitLab as the repository for your docker images.

You can also use DockerHub. I also wrote an article How to upload a docker image to the Docker Registry on Docker Hub.

How to deploy an application?

There are typically two ways to deploy an application:

• Help the container and perform a build, push and pull of the panel with the new version of the application.
• Classically, run some deploy script that performs a git pull and other actions.

Operating without containers

Deploy your application depending on how you have set up your environment to run the application.
You don't need to have the application in containers to run the application at all. It is even possible to run multiple PHP applications on a single server, including different versions of PHP, and keep the applications separate from each other, thus ensuring basic security. Of course, you need to know how to Linux web server to install.
To deploy, you can then use, for example, the Deployer tool. Or you can write your own script.

Application deployment in docker

If you will be running the application in a container, the deploy just consists of downloading and new image and dropping the container from it. Typically some migration scripts need to be run as well. Remember to keep your data on volume so that your data is persistent.

Simple GitLab CI example

An example of a simple .gitlab-ci.yml file to verify that CI works on Gitlab might look like this:

stages:
  - build
  - test
  - deploy

build_project:
  stage: build
  script:
    - echo "Building the project..."

run_tests:
  stage: test
  script:
    - echo "Running tests..."

deploy_to_production:
  stage: deploy
  script:
    - echo "Deploying to production..."
  environment:
    name: production
    url: https://your-production-url.com

Commit the file to the project and then the pipeline should run.

GitLab CI and variables

To function, you will need to pass variables to the jobs in the pipeline. In GitLab CI/CD, you can pass variables in several different ways.

Remember to be mindful of the security of sensitive information and variables. In the case of sensitive data, it's a good idea to use GitLab's features to protect that data. For example, only allow them to be used in privileged situations.

Defining variables in repository settings

To store environment variables (ENVs) and access keys securely in GitLab, there are "variables" and "CI/CD environment variables", which can be set in the GitLab interface itself and are not part of your application's source code. This helps protect sensitive information and allows them to be used within the CI/CD pipeline.
In the GitLab interface itself (in the repository settings), you can define environment variables that will be accessible in your .gitlab-ci.yml. These settings can be generic for the whole project or can be specific to each branch, tag.
To add variables to your GitLab repository, go to project settings -> "Settings" -> "CI/CD" -> "Variables". Here you can define variables of different types (such as environment variables, file type variables, and more).
These variables can be used within your CI/CD pipeline using the classic bash syntax e.g. $VARIABLE_NAME.

For example, you can pass an ssh key in this way so that you can then connect via ssh to the server when you deploy your application.

Defining variables directly in .gitlab-ci.yml

You can pass variables directly in the .gitlab-ci.yml configuration file.

yaml

stages:
  - build

variables:
  MY_VARIABLE: "value"

build_project:
  stage: build
  script:
    - echo $MY_VARIABLE

In this case, the MY_VARIABLE variable is defined directly in the .gitlab-ci.yml file and is used in the build_project job.

Passing variables via scripts

If you need to dynamically change the value of variables depending on conditions, you can pass them through scripts in the CI/CD pipeline

.
yaml

Stages:
  - build

build_project:
  stage: build
  script:
    - export MY_VARIABLE="value"
    - echo $MY_VARIABLE

This example shows the use of export within a script to set the MY_VARIABLE variable.

Choosing another image in GitLab CI

Can define globally which docker image to use. But in different jobs you will be doing different exits, so you will want to use different images.
For example, if you need an ssh client to connect to a server using ssh, you will need an image where the ssh client is. For a build docker image you will need an image where the utilities for the build docker image are.

The definition that the image docker:19.03.13 is to be used.

build_image:
  before_script:
    - ''
  only:
    - main
  stage: build
  image: docker:19.03.13
  services:

It also defines that this job will only run when commit to main.

Connecting via ssh

When deploying the application, it will probably need to connect to the server, ideally using ssh. How to do this? First of all, the best option is to use SSH keys. Add the contents of your SSH private key as a secret variable to GitLab. This will ensure that the key is not publicly visible.

In order to have an application to log in to ssh, we use the image kroniak/ssh-client, which contains the ssh client. We then prepare the environment so that the configuration for ssh is as we need it, including the ssh key. This allows you to connect to the server securely and conveniently. And then you just need to run the commands on the server...

Configure the connection using the SSH key in .gitlab-ci.yml:

deploy:
  before_script:
    - ''
  services: []
  stage: deploy
  image: kroniak/ssh-client
  only:
    - main
  script:
    - mkdir ~/.ssh/
    - echo "$SSH_KEY" | tr -d '\r' > ~/.ssh/id_rsa
    - echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config
    - chmod 600 ~/.ssh/id_rsa
    - export SSH_AUTH_SOCK && export SSH_ASKPASS=ssh-askpass && eval `ssh-agent -s` && ssh-add ~/.ssh/id_rsa
    - "scp docker-compose.prod.yml user@test3.xeres.cz:app"

In this case, the docker-compose.prod.yml file is copied to the server via ssh.

Keep in mind that storing a private key in the CI/CD pipeline can be risky and should be used with caution. It is recommended to use special keys with limited permissions and create a user account with limited privileges. Such an account will only be used for CI/CD purposes.

Deploying a Docker container

The steps for deploying a Docker container via GitLab CI/CD to a server are:

1. Create a Docker image: Make sure you have a Dockerfile that describes your application and its environment. Create a Docker image using the docker build command. For example: docker build -t your-image-name:tag.
2. Push the image to the registry: If you are using external Docker registries (for example, Docker Hub), log in using the docker login command.
3. On the server, download a new version of the image, shut down the old container, and start the container from the new image.

The following example shows the steps of building, logging into the registry and pushing the image.

build_image:
  before_script:
    - ''
  only:
    - main
  stage: build
  image: docker:19.03.13
  services:
    - docker:19.03.13-dind
  script:
    - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN registry.gitlab.com
    - docker build ./ -t registry.gitlab.com/$CI_PROJECT_NAMESPACE/$CI_PROJECT_NAME:app
    - docker push registry.gitlab.com/josef.jebavy/example-symfony-devstack-cicd:app

	

After adding and pushing the application changes to the repository, the CI/CD pipeline will run and automatically build a docker image with the new application version and push to the registry.
This job uses docker:19.03.13 for the build image, which contains tools for building docker images.

The following example shows the steps of logging into the server using ssh, copying the new configuration docker-compose.prod.yml, downloading the new image with the new application. Shutting down the old container, starting the container from the new image and running the migration scripts.

Deploy:
  before_script:
    - ''
  services: []
  stage: deploy
  image: kroniak/ssh-client
  only:
    - main
  script:
    - mkdir ~/.ssh/
    - echo "$SSH_KEY" | tr -d '\r' > ~/.ssh/id_rsa
    - echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config
    - chmod 600 ~/.ssh/id_rsa
    - export SSH_AUTH_SOCK && export SSH_ASKPASS=ssh-askpass && eval `ssh-agent -s` && ssh-add ~/.ssh/id_rsa
    - "scp docker-compose.prod.yml user@test3.xeres.cz:app"
    - ssh user@test3.xeres.cz "docker login -u gitlab-ci-token -p $CI_JOB_TOKEN registry.gitlab.com && docker pull registry.gitlab.com/josef.jebavy/example-symfony-devstack-cicd:app"
    - ssh user@test3.xeres.cz "cd app && docker compose -f docker-compose.prod.yml stop web && docker compose -f docker-compose.prod.yml rm web -f && docker compose -f docker-compose.prod.yml up -d "
    - ssh user@test3.xeres.cz "cd app && docker exec -d test-app-web bin/console doctrine:database:create --if-not-exists && docker exec -d test-app-web php bin/console doctrine:migrations:migrate --no-interaction"

GitLab CI example

For a complete example of a GitLab CI configuration where automated tests, build application and deploy to production, click here https://gitlab.com/josef.jebavy/example-symfony-devstack-cicd/. The demo also includes server configuration, which you can do automatically using Ansible. If you do a fork of my project, you can also try it yourself and deploy to your server.

Video tutorial

Here is a step-by-step detailed sample video tutorial on how to get GitLab CI working:

Operating a project in a docker container

How to run a project in docker containers:

Automatic tests

GitLab CI/CD and automated tests:

Pipeline

Gitlab CI/CD and pipeline settings in gitlab-ci.yml:

Deploy application to server

GitLab CI/CD settings to deploy the application to the server:

Server Configuration

GitLab CI/CD server configuration to run the application in a container and setup for automatic deploy using GitLab CI:

Automation

Automation will do a lot of the work for you, saving you valuable time. You will also reduce human error. At a minimum, I recommend having tests and running those automatically when you commit to a git repository. This way with every source code modification you will be sure to test your application.

Links

• https://bogomolov.tech/gitlab-ci-cd-symfony/
• https://medium.com/@zakariae/setting-up-symfony-continuous-deployment-using-rancher-274838fcf632