CI Runners for external projects

Owned by Pierrick Bouvier
Last updated: Apr 28, 2022
2 min read

Since Azure started enabling arm64 virtual machines (april 2022), we have been investigating how to use them to implement CI/CD for external projects, using Github Actions/Azure pipelines and self-hosted runners/agents.

We need to solve three different challenges:

  1. Sharing: how to share runners between different projects/organizations?

  2. Isolation: ensure every job gets a clean and secure environment

  3. Scaling: how to dynamically provision runners as needed?

Introduction

First, let’s present how Github actions/Azure pipelines work with a self-hosted runner.

1. Developer pushes its code 2. Github infra triggers a ci job on runner 3. The runner executes command locally and returns result

Challenges

Sharing

A runner (or azure agent) is currently registered only for one project or organization. It implies that that you can’t reuse the same runner for different organizations/projects.

⚠️ Potential solution

Run one runner for every project/organization we plan to support. This does not scale very well, and does not ensure a single machine runs a single job at a time (several runners → several jobs running at the same time on the same machine).

Isolation

It’s important that two jobs do not interfere with each other. Main reasons are reliability and security.

  • Reliability: if a given job breaks machine or leaves traces, it should not impact next jobs.

  • Security: if a job contains malicious code, it should not impact next jobs.

❌ No solution

So far, there is no way to ensure this. Github goes as far as recommending to avoid it.

About self-hosted runners - GitHub Docs

Github-hosted runners (and Azure-hosted), on the opposite, offers this isolation, but there is no arm64 runner available yet.

Scaling

To avoid huge costs on our cloud infrastructure, we need to be able to start/stop VM on demand, when a new job asks to run.

❌ No Azure pipeline solution

On Azure, you can use “Scale sets” to run your pipeline. In this case, Azure takes care of everything for you, which is great.

A scale set is attached to a paid subscription, which ensures payment. Alas, it’s not possible to share a scale set (nor subscription) between different projects/organizations.

❌ No Github solution

On Github, autoscaling is supported through two different projects:

Actions-runner-controller relies on Kubernetes, which means it uses containers. Alas, Azure Kubernetes is not supported yet for Windows on Arm. Worker nodes based on ARM processors · Issue #1613 · Azure/AKS

The second option, based on AWS and full VM, does not support Azure.

It means we have no solution yet for a Windows On Arm/Azure environment.

Conclusion

From what we saw, there is no solution yet to solve our 3 challenges.

Those challenges are already handled by Azure/Github runners, but there is no support for arm64/windows yet GitHub - actions/runner-images: GitHub Actions runner images .

Our missing bits are:

  • Sharing: no way to share runners between organizations/projects

  • Isolation: no way to run an isolated environment with self-hosted runners

  • Scaling

    • Azure: no way to share a scale set between organizations/projects

    • Github: no solution for Azure/Windows on Arm. Needs container support (+ base images) for Windows on Arm.