Skip to main content

🤖 CI/CD

Requirements​

warning

This will nest a full Ludus install for every pipeline in an existing Ludus server. Going more than 1 layer deep of nested virtualization is not supported.

To set up a CI/CD runner for Ludus development you must meet the following requirements:

  1. A functional, fast, Ludus server with at least 128GB of free RAM, 1TB of free disk space, and as many fast cores as you can get
  2. The debian-13-x64-server-template must be built
  3. Root access to the Ludus server
  4. A Gitlab account with the ability to create a runner token (gitlab.com or self-hosted)
  5. Network access from the Ludus server to the Gitlab instance/gitlab.com

Setup​

To setup the CI/CD runner and template follow these steps:

  1. Build the debian-13-x64-server-template Ludus template.

  2. Create a GitLab Runner with the tag ludus-proxmox-runner-parallel. Do not check Run untagged jobs. Provide the runner token as an env variable to the bootstrap-ci-host.sh script

  3. Clone the Ludus repository on the Ludus host and run the CI bootstrap as root:

LUDUS_ADMIN_API_KEY=JD... GITLAB_RUNNER_TOKEN=... ./ludus-server/ci/bootstrap-ci-host.sh

You can also provide Proxmox credentials directly, and modify other settings via ENV variables to match your environment:

PROXMOX_USERNAME=root@pam PROXMOX_PASSWORD='...' PROXMOX_VM_STORAGE_POOL='<storage pool>' PROXMOX_VM_STORAGE_FORMAT=raw NER_TOKEN=... ./ludus-server/ci/bootstrap-ci-host.sh

When the bootstrap finishes, you will see debian-13-x64-server-ludus-ci-template, the protected ci-seed-* templates, the cluster VMs, and the build VM in the Proxmox web UI. The existing GitLab Runner will be configured to use the Ludus custom executor scripts in /opt/ludus/ci. CI VMs default to 250GB root disks; set CI_VM_DISK_SIZE=300G or another G value before running the bootstrap if your template/range workload needs more space.

The cluster VMs are bootstrapped with lae.proxmox as a nested two-node Proxmox cluster before their clean snapshots are taken. VMID 1005 is node ludus, VMID 1006 is node ludus2, VM disk storage is the Ceph RBD datastore ceph, and ISO storage is the CephFS datastore cephfs.

Now that CI is setup and configured, any commits that are pushed to the Ludus project will build and test as appropriate.

Tags​

The CI system is set up to run the appropriate tests depending on what part of the code base has been modified. However, sometimes you want to override the defaults. To manually control the CI pipeline, you can add "tags" to the final commit message before a push. To use these, simply include one or more of the "tag" strings in your commit message, including the brackets.

The available tags are listed below:

  • [skip ci] - this tag skips all CI jobs
  • [skip build] - skips the documentation build and the binary build stages
  • [build docs] - force a documentation build
  • [build pages] - force a documentation build and pages deploy
  • [full build] - run every step of the CI pipeline, no matter how small the change to the code base
  • [manual] - only run the documentation build (if docs have changed) and binary build, then start a manual testing job on a pre-built CI VM
  • [client tests] - test basic client commands that do not deploy templates or ranges
  • [template tests] - run a template build and wait for all templates to complete building
  • [range tests] - run a range deploy and wait for it to succeed. This uses the simple-domain.yml range config.
  • [post-deploy tests] - runs all tests related post-deployment tasks (testing mode, allowing and denying domains and IPs, powering on and off a VM, adding and removing users, etc)
  • [testing-mode tests] - runs tests to determine if testing mode functions properly
  • [ansible tests] - runs tests to determine if ansible features function properly
  • [user tests] - runs tests to determine if the functions related to user management function properly
  • [template tests] - runs tests related to adding, building, and removing custom templates
  • [start-at templates] - run all test starting at the template builds
  • [start-at range-admin] - run all test starting at the deployment of the admin user range
  • [start-at post-deploy-admin] - run all test starting after the admin range has deployed
  • [start-at range-user] - run all test starting at the deployment of the standard user range
  • [start-at post-deploy-user] - run all test starting after the deployment of the standard user range
  • [start-at integration] - just run the final integration test

Releases​

Any time a version tag is created in Gitlab, two additional CI jobs are added to the pipeline: upload and release. These jobs are manually triggered (you must click the play button in the pipeline) and upload the compiled binaries to the package registry as well as create the actual release. If you use conventional commits (perhaps created with koji), then git-cliff will automatically generate a change log for the release.

Manual CI VM Setup​

Run these commands on a Debian 13 VM, then power it off and save it as a template

hostname ludus-ci-debian-13

# Resize the disk by hand if needed (bootstrap uses 250GB by default)
fdisk /dev/vda1
p
d
n
1
2048
[End]
N
w

resize2fs /dev/vda1

# Install Go
apt install curl wget ca-certificates
wget https://go.dev/dl/go1.25.1.linux-amd64.tar.gz
rm -rf /usr/local/go && tar -C /usr/local -xzf go1.25.1.linux-amd64.tar.gz
# This is required since gitlab-runner ignores .bashrc
echo 'export PATH=$PATH:/usr/local/go/bin' >> /etc/profile


# Add the gitlab-runner user and allow them to sudo
useradd -m -s /bin/bash -c "Gitlab Runner" gitlab-runner 
# This breaks gitlab-runner, remove it
rm /home/gitlab-runner/.bash_logout
echo 'gitlab-runner ALL=(ALL) NOPASSWD: ALL' >> /etc/sudoers

# Install needed components
curl -L 'https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh' | bash
curl -s 'https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh' | bash
apt install gitlab-runner git git-lfs build-essential vim tmux htop jq python3-debian

# Install node/yarn for documentation building
curl -fsSL https://deb.nodesource.com/setup_21.x | bash - && apt-get install -y nodejs
npm install --global yarn

# Helpful to auto-load the key on login for troubleshooting
echo 'if [ -f /opt/ludus/ci/.apikey-admin ]; then export LUDUS_API_KEY=$(cat /opt/ludus/ci/.apikey-admin); fi' >> /home/gitlab-runner/.bashrc
echo 'if [ -f /opt/ludus/ci/.apikey-user ]; then export LUDUS_API_KEY=$(cat /opt/ludus/ci/.apikey-user); fi' >> /home/gitlab-runner/.bashrc

# Warm the caches
su gitlab-runner -
cd /tmp
git clone https://gitlab.com/badsectorlabs/ludus

cd ludus/ludus-server
GOOS=linux GOARCH=amd64 go build -trimpath -ldflags "-s -w"

cd ../ludus-client
git clone https://github.com/zimeg/spinner
cd spinner && git checkout unhide-interrupts && cd .. && go mod edit -replace github.com/briandowns/spinner=./spinner
GOOS=linux GOARCH=amd64 go build -trimpath -ldflags "-s -w"
GOOS=linux GOARCH=arm64 go build -trimpath -ldflags "-s -w"
GOOS=darwin GOARCH=amd64 go build -trimpath -ldflags "-s -w"
GOOS=darwin GOARCH=arm64 go build -trimpath -ldflags "-s -w"
# The forked spinner library doesn't compile for windows, so switch back to the original
go mod edit -dropreplace=github.com/briandowns/spinner
GOOS=windows GOARCH=amd64 go build -trimpath -ldflags "-s -w"
GOOS=windows GOARCH=386 go build -trimpath -ldflags "-s -w"
GOOS=windows GOARCH=arm64 go build -trimpath -ldflags "-s -w"

cd ../docs
yarn install
yarn build