CI Runners for external projects

Owned by Pierrick Bouvier
Last updated: Apr 28, 2022
2 min read

Since Azure started enabling arm64 virtual machines (april 2022), we have been investigating how to use them to implement CI/CD for external projects, using Github Actions/Azure pipelines and self-hosted runners/agents.

We need to solve three different challenges:

  1. Sharing: how to share runners between different projects/organizations?

  2. Isolation: ensure every job gets a clean and secure environment

  3. Scaling: how to dynamically provision runners as needed?

Introduction

First, let’s present how Github actions/Azure pipelines work with a self-hosted runner.

1. Developer pushes its code 2. Github infra triggers a ci job on runner 3. The runner executes command locally and returns result

Challenges

Sharing

A runner (or azure agent) is currently registered only for one project or organization. It implies that that you can’t reuse the same runner for different organizations/projects.

⚠️ Potential solution

Run one runner for every project/organization we plan to support. This does not scale very well, and does not ensure a single machine runs a single job at a time (several runners → several jobs running at the same time on the same machine).

Isolation

It’s important that two jobs do not interfere with each other. Main reasons are reliability and security.

  • Reliability: if a given job breaks machine or leaves traces, it should not impact next jobs.

  • Security: if a job contains malicious code, it should not impact next jobs.

❌ No solution

So far, there is no way to ensure this. Github goes as far as recommending to avoid it.

https://docs.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners#self-hosted-runner-security

Github-hosted runners (and Azure-hosted), on the opposite, offers this isolation, but there is no arm64 runner available yet.

Scaling

To avoid huge costs on our cloud infrastructure, we need to be able to start/stop VM on demand, when a new job asks to run.

❌ No Azure pipeline solution

On Azure, you can use “Scale sets” to run your pipeline. In this case, Azure takes care of everything for you, which is great.

A scale set is attached to a paid subscription, which ensures payment. Alas, it’s not possible to share a scale set (nor subscription) between different projects/organizations.

❌ No Github solution

On Github, autoscaling is supported through two different projects:

Actions-runner-controller relies on Kubernetes, which means it uses containers. Alas, Azure Kubernetes is not supported yet for Windows on Arm.

The second option, based on AWS and full VM, does not support Azure.

It means we have no solution yet for a Windows On Arm/Azure environment.

Conclusion

From what we saw, there is no solution yet to solve our 3 challenges.

Those challenges are already handled by Azure/Github runners, but there is no support for arm64/windows yet .

Our missing bits are:

  • Sharing: no way to share runners between organizations/projects

  • Isolation: no way to run an isolated environment with self-hosted runners

  • Scaling

    • Azure: no way to share a scale set between organizations/projects

    • Github: no solution for Azure/Windows on Arm. Needs container support (+ base images) for Windows on Arm.