Gitlab

Besides using Jenkins, you can use gitlab’s integrated CI pipelines.

You can either use one of the shared runners (Linux only) or set up a CI runner on a machine (it can be any machine, for instance one of your own machines, or a VM hosted in the CI cloudstack).

Enabling CI on a gitlab project

• In your project’s General settings, in the section Visibility, project features, permissions, search for CI/CD, and check the feature is enabled. Once you enabled the feature, remember to click on Save changes.

• Then a CI / CD tab should now appears in the settings.

• In this CI / CD tab, in the Runners section, runners appear in two columns

  • Specific runners on the left lists runners that have been registered for this project and runners available from other projects you are member of;
  • Shared runners on the right allows you to Enable shared runners for this project (check box), and lists available shared runners.

• On top of the Specific runners columns, instructions and keys are given to register new runners for this project.

To perform the actual build, you can either use one of the shared runners available (Linux only), or install gitlab-runner on a machine and register this new runner on Gitlab (gitlab-runner can be run on Linux, Windows or Mac OS).

Using shared runners (Linux only)

Once the option Enable shared runners for this project is checked in the settings for CI / CD runners, the three following docker runners are available:

• #4663 (xxjFYZX3) Inria CI gitlab-small (1 CPU, 2048 MB RAM), tags: ci.inria.fr, linux, small
• #4664 (J-gEBwyb) Inria CI gitlab-medium (2 CPUs, 4096 MB RAM), tags: ci.inria.fr, linux, medium
• #4665 (2mk6rsew) Inria CI gitlab-large (4 CPUs, 8096 MB RAM), tags: ci.inria.fr, linux, large

Each job is run inside a dedicated virtual machine with 16Go of available hard disk space.

For the time being, there are:

• 8 slots for gitlab-small
• 4 slots for gitlab-medium
• 11 slots for gitlab-large

Although they are running inside a container, the jobs do have access to the docker socket at /var/run/docker.sock. This means that you can:

• build docker images and push them to your project registry (hosted on the gitlab server)
• use docker-compose or any other tool to deploy your build/test environment

The jobs have total access to the underlying docker engine (they can even run privileged containers), however you should be aware that:

• for security reasons, each job is isolated it its own virtual machine, with its own docker engine (thus you have only access to the containers of the current job)
• when binding external volumes (docker run -v HOSTPATH:CONTAINERPATH), the host path refers to the filesystem of the virtual machine (rather than the filesystem of the container running the job). If you need to bind an external volume for your jobs, then you should only use paths inside /builds or /tmp (because these are the only directory that are mounted from the VM filesystem).
• when publishing ports (docker run -p HOSTPORT:CONTAINERPORT), the host port is binded on the virtual machine (rather than on the container running the job). It can be reached at 172.17.0.1 which is the actual ip of the host interface on the docker0 bridge

You are welcome to run your CI jobs on these runners. See Configuring CI tasks section for configuring pipelines via the .gitlab-ci.yml file of your repository.

Caching

In order to speed up the builds, the downloaded dependencies need to be cached for subsequent builds.

As of today we are not using the native cache provided by Gitlab. This feature is very fine for running a personnal runner, but it is unclear how useful it will be for a heavily shared runner knowing that:

• it has to be explicitly configured for every job. Every developer has to worry about where the dependencies is stored in container and properly configure the paths to be cached in the .gitlab-ci.yml. Also there is no very useful feedback indicating whether the configuration is correct or not.

• it is unclear how to efficiently manage the storage space For distributed setups, gitlab-runner provides a distributed cache feature. that uses an object storage service to host the cached content, but nothing on how to manage the space (which actually makes sense because it is developed by a company which sells cloud services and they certainely won’t mind if you use too much storage space).

Our approach is to cache as much as possible the result of outgoing HTTP requests. Then if needed, we will deploy a distributed cache to support the remaining corner cases.

Most of the traffic is already transparently cached and does not require any user action. This especially covers (the list is not exhaustive): images from the docker hub, packages from debian, ubuntu, centos, fedora, centos, alpine, pip, conda.

Other traffic require explicit configuration by the user, for the moment the only such case is Maven (see below).

Transparent cache

The outgoing HTTP streams are redirected by the runners to our local caching server. Which is able to process:

• all outgoing plain text HTTP traffic on port 80 (unencryted)
• all outgoing HTTPS traffic on port 443 originated by applications linked with openssl or gnutls

The redirection is implemented with iptables NAT rules inserted in the container network namespace. For HTTPS traffic we also need to load the CA certificate of the proxy. This done by preloading a library (with the LD_PRELOAD environment variable), which intercepts a few openssl and gnutls api calls.

These changes are injected through the OCI runtime API (when docker call runc to start the container). The docker daemon is not aware about their presence. Thus when you build docker images, none of these changes will be present in the final images (although they are present during the build).

Disabling transparent caching

If you have issues with HTTPS, you may unset the LD_PRELOAD variable before calling the command that fails (this may be useful if you suspect that the preloaded lib interferes with the build).

When starting our own containers with the Docker API, you can add --runtime runc to your docker run/docker build command (to swich back to the default OCI runtime). This will completely disable caching for HTTP and HTTPS in these containers.

Maven mirror

There is a caching mirror of the central Maven repository hosted at https://docker-ci-svc.inria.fr/https/repo.maven.apache.org/maven2/. We could not find a way to have the jobs use it transparently an non-disruptingly, so you have to configure it explicitely.

The recommended way to use it in your builds is to prepend $CI_MAVEN_OPTS in the arguments for the mvn command in your .gitlab-ci.yml (or elsewhere). Eg:

script:
  - mvn $CI_MAVEN_OPTS package

You can also use it in a Dockerfile (the variable is present during the build, but won’t remain in the final image because it is injected at the OCI runtime layer):

RUN mvn $CI_MAVEN_OPTS package

This $CI_MAVEN_OPTS expands to a --global-settings argument which loads a settings file that looks like:

<settings>
  <mirrors>
    <mirror>
      <id>ci-central</id>
      <mirrorOf>central</mirrorOf>
      <url>https://docker-ci-svc.inria.fr/https/repo.maven.apache.org/maven2</url>
    </mirror>
  </mirrors>
</settings>

Caution: be aware that loading this file overrides maven’s globals settings (${maven.conf}/settings.xml, which are usually located at /etc/maven/settings.xml or /usr/share/maven/conf/settings.xml depending on your distribution). In most cases this is not a concern because this global settings file is empty by default. You may however encounter side-effects when using docker image which uses a customised version.

Troubleshooting

If you have a problem that you are unable to fix, you can report it to the ci-support team.

Setup for Gitlab Pages

If you want to generate a static web site using Gitlab Pages, then your job must comply with some rules:

• the job name must be pages
• the public/ subdirectory must be declared as an output artifact
• the job must output the web site content into the public/ subdirectory

Also it is a good practice to restrict this job to be run only for a specific branch (eg. the master branch)

---
pages:
    image: busybox
    artifacts:
        paths:
        - public
    script:
     - mkdir public
     - echo '<html><head><title>42</title></head><body><h2>What do you get if you multiply six by nine?</h2></body></html>' > public/index.html
    only:
        - master

Building and pushing Docker images

The docker socket is accessible from inside the jobs, therefore it is possible to build and push docker images.

To push an image to your gitlab registry, you need to prefix its name with $CI_REGISTRY_IMAGE/ (which will be expanded to the address of your project registry hosted on the gitlab server).

Before pushing you need to authenticate as user gitlab-ci-token with password $CI_JOB_TOKEN.

A basic example is given in the two files below. The pipeline runs two jobs:

• stage1 builds the hello image using the Dockerfile and pushes it to the project registry. This image contains a /bin/greeter command that displays Hello World!.
• stage2 runs a job using the hello image built in stage1. The job executes the /bin/greeter command.

Dockerfile

FROM busybox

RUN (echo "#!/bin/sh"; echo "echo 'Hello World!'") > /bin/greeter
RUN chmod 0755 /bin/greeter

.gitlab-ci.yml

---
stages: ["stage1", "stage2"]

build-hello:
    stage: "stage1"
    tags:
        - ci.inria.fr
        - small
    image: docker:stable
    script:
        - docker build -t "$CI_REGISTRY_IMAGE/hello" .
        - docker login -u gitlab-ci-token -p "$CI_JOB_TOKEN" "$CI_REGISTRY"
        - docker push "$CI_REGISTRY_IMAGE/hello"


run-hello:
    stage: "stage2"
    tags:
        - ci.inria.fr
        - small
    image: "$CI_REGISTRY_IMAGE/hello"
    script: 
     - /bin/greeter

Changelog

This section lists all the changes in the configuration of the shared runners since their initial deployment in production.

• 2023-02-14

  • upgrade to docker 20.10.23 and linux 5.15.92

• 2022-03-31

  • increase the size of /dev/shm to half the VM memory

    By default gitlab-runner jobs are run with a 64Mo /dev/shm. This is dockerd’s default and this may be too small for certain jobs.

    The new config resizes it to half the total VM memory (eg: a job running in a VM with 4Go RAM will have a 2Go /dev/shm). This new value matches the default /dev/shm size for native process on most linux distributions.

• 2022-03-11

  • enable dockerd experimental features

    This allows using docker build --squash to reduce the size of the generated images. However keep in mind that experimental features may not be stable and may be dropped at any time.

• 2022-03-08

  • initial deployment

Installing runners (Linux, Windows or Mac OS)

gitlab-runner can run anywhere: you can use a virtual machine on ci.inria.fr (Linux, Windows or MacOS) to host your GitLab runner.

See:

• https://ci.inria.fr/doc/page/web_portal_tutorial/ to create a slave on Inria’s CI platform and access it (you can ignore the jenkins related parts)
• https://docs.gitlab.com/runner/install/ for the official documentation to install the runner on the vm(s) you’ve created.

Installation example on a GNU/linux slave from ci.inria.fr (“ci” user account)

sudo wget -O /usr/local/bin/gitlab-runner https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-linux-amd64
sudo chmod +x /usr/local/bin/gitlab-runner
sudo gitlab-runner install --user=ci --working-directory=/builds
sudo gitlab-runner start
sudo gitlab-runner status # should return "service is running"

Installation example on a macOS slave from ci.inria.fr (“ci” user account)

Limitations on macOS : The service needs to be installed from a Terminal window logged in as your current user (i.e., “ci” user account).

sudo curl --output /usr/local/bin/gitlab-runner https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-darwin-amd64
sudo chmod +x /usr/local/bin/gitlab-runner
# Run the following commands as the "ci" user
gitlab-runner install --working-directory /builds
gitlab-runner start
gitlab-runner status # should return "service is running"

Installation example on a Windows slave from ci.inria.fr (“ci” user account)

# Follow https://docs.gitlab.com/runner/install/windows.html then when installing the service prefer the following to use the existing "ci" user account
gitlab-runner install -u ".\ci" -p "ci" -d "C:\Users\ci"
gitlab-runner start
gitlab-runner status # should return "service is running"

Register a runner in your project

On the virtual machine where you installed the gitlab CI runner, run (as root or sudo if the gitlab-runner program has been installed with sudo, remove sudo from the following if not) :

$ sudo gitlab-runner register 

Use the gitlab URL and the registration token found in the “Pipelines” page when asked.

In order to answer when asked

Please enter the gitlab-ci tags for this runner (comma separated):

it is important to know that “gitlab-ci tags” are NOT git tags. The tags you enter here are useful to specify conditional execution in the configuration file .gitlab-ci.yml.
See :

• https://docs.gitlab.com/ce/ci/yaml/#tags
• and https://docs.gitlab.com/ce/ci/yaml/#only-and-except

for more details.

The next question

Whether to run untagged builds [true/false]: 

becomes clear when the “gitlab-ci tag” concept is understood.

When asked

Please enter the executor:

you should answer shell (or docker but then see the related subsection below)

At the end of this step, your runner should appear in the Settings > Pipelines tab of your project.

Configurating CI tasks

Finally, configure the tasks to run by creating a .gitlab-ci.yml file at the root of your project.
Follow the official documentation at https://docs.gitlab.com/ce/ci/yaml/ to create this file.

Note that shared runners only run jobs when at least one tag matches: remember to tag your job with one of the tags ci.inria.fr or linux or small, medium or large for the job to be executed on shared runners. For instance, to target the small shared runner:

my-job:
  tags:
    - ci.inria.fr
    - small

Using a docker executor

It has been reported that there are DNS issues with docker running on the INRIA CI’s VMs (due to bad interaction between dnsmasq and docker, see https://stackoverflow.com/questions/49998099/dns-not-working-within-docker-containers-when-host-uses-dnsmasq-and-googles-dns). This can be solved by adding network_mode = "host" in the configuration. Eg. /etc/gitlab-runner/config.toml:

concurrent = 1
check_interval = 0

[[runners]]
name = "ci.inria"
url = " [ https://gitlab.inria.fr/ | https://gitlab.inria.fr/ ] "
token = "..."
executor = "docker"
[runners.docker]
network_mode = "host"
tls_verify = false
image = "alpine:latest"
privileged = false
disable_cache = false
volumes = ["/cache"]
shm_size = 0
[runners.cache]

Examples

If you look for “real life” gitlab-ci examples, please visit this dedicated group gitlabci_gallery. It contains several subgroups and git repositories showing some interesting key features of gitlab-ci and possible integrations with external tools/platforms (ci.inria.fr, terraform, github, a supercomputer, etc).