Setting up an LLVM Build Master

Creating an LLVM buildbot master requires you to install the correct version of buildbot and its dependencies (which is not the most recent ones, due to incompatibilities on the LLVM side), and the LLVM buildbot integration, aka. Zorg. You should also make sure you have git installed and in the path, otherwise the buildmaster will not work correctly.

The buildmaster will use two ports, one for the web server and one for communication with the workers. We will use the convention 801X and 901X. If you're setting up your buildmaster in a docker container, make sure you forward these ports when you start it:

$ docker run -p 801X:801X/tcp -p 901X:901X/tcp [...]

Installing Zorg

The first thing you need to do is to install Zorg. That's because you'll need it to tweak your buildbot installation below. If this is a testing/staging buildmaster installation, I recommend you to install on your home directory, as it makes it a lot easier to edit the configurations, inspect the logs and restart the server. Otherwise, it's better to create a buildbot user and an environment on something like /var/buildmaster or similar.

Get Zorg

The best way is to check it out from the official repositories:

$ git clone https://github.com/llvm/llvm-zorg.git

Install Zorg

Buildbot is a Python based program, so it'll need all classes in Zorg available in the search path. You can either add the llvm-zorg directory to your search path (PYTHONPATH).

Also make sure you install all of the dependencies:

$ pip3 install -r requirements.txt

With this, all changes, repo updates and branch moves will automatically be reflected on your buildmaster.

On MacOS, a rust compiler is needed and some compiler flags need to be passed for the installation of some packages to succeed:

$ brew install rust
$ env LDFLAGS="-L$(brew --prefix openssl@3)/lib" CFLAGS="-Wno-error=implicit-function-declaration -I$(brew --prefix openssl@3)/include" pip3 install -r requirements.txt

NOTE: it's better to use virtualenv to avoid conflicts between system's python packages and those installed with pip.

Installing Buildbot for LLVM

LLVM buildbot setup is different on different platforms.  This has proven to work in a Ubuntu Bionic docker container, and your mileage on other setups may vary.  You need python 3.x and a version of buildbot as close as possible to what the production buildmaster is running:

$ pip3 install --user buildbot buildbot-worker buildbot-www buildbot-waterfall-view buildbot-console-view buildbot-grid-view

Successfully installed ...
Segmentation fault (core dumped)

$ rm -rf ~/.local/

On MacOS, the latest buildbot version didn't work, but version 2.8.4 worked fine (04/12/2023):

$ pip3 install buildbot==2.8.4 buildbot-worker==2.8.4 buildbot-www==2.8.4 buildbot-waterfall-view==2.8.4 buildbot-console-view==2.8.4 buildbot-grid-view==2.8.4

Now, you need to create a build dir:

$ buildbot create-master ~/buildmaster
NOTE: If you get errors finding packages that pip has successfully installed, make sure you have $HOME/.local/bin in your PATH and $HOME/.local/lib/python3.6/site-packages in PYTHONPATH.

This will create the directory and populate it with an empty SQLite database, a sample master config and the default buildbot.tac configuration.

NOTE: If you get errors like "OSError: [Errno 13] Permission denied: '/..../decorator.py'", it means that package is not installed system-wide. Use "sudo pip install <package>" to get it to work.

NOTE2: Avoid installing any other package (especially twisted, buildbot, etc.) as root. This would break other people's bots if you're not in a container. Just install the ones missing.

NOTE3: On MacOS, it was necessary to upgrade package pyOpenSSL (latest: 23.3.0) and downgrade markupsafe (2.0.1).

Adding Zorg's config to buildmaster

Symlink

• ln -s ~/llvm-zorg/buildbot/osuosl/master/master.cfg ~/buildmaster/master.cfg
• ln -s ~/llvm-zorg/buildbot/osuosl/master/config ~/buildmaster/config

You'll have to change several files to adjust the configuration to your environment:

master.cfg

In the "Workers" section, change:

c['protocols'] = { 'pb' : { 'port' : 901X } } # X is your master/worker number (same as 801X for the web interface)

On "PROJECT IDENTITY", change:

c['title'] = "Linaro LLVM - <Your Name>"
c['titleURL'] = "http://masterX.llvm.validation.linaro.org/" # X is your master number (same as above)
c['buildbotURL'] = "http://masterX.llvm.validation.linaro.org/" # X is your master number (same as above)

Don't forget to change the port for c['www'] if necessary. Also note that the URLs above should end with a '/'.

And on "DB URL", set it to a local SQLite:

c['db'] = { 'db_url' : 'sqlite:///state.sqlite' }

config/workers.py

This file will have a section on passwords. The official master queries the database, we don't need that and can use a dummy password for internal purposes. Pick one and change create_worker to:

def create_worker(name, *args, **kwargs):
 password = 'DummyPasswd' # Choose any random password, doesn't matter.
 return worker.Worker(name, password=password, *args, **kwargs)

When starting workers to connect here, you'll need to update the buildbot.tac file (or use the right password on the command-line).

config/builders.py

Remove *-sphinx-*  builders if they cause problems on buildmaster startup.

In this case, the publish-lnt-sphinx-docs builder also needs to be removed, from getLntSchedulers(), in config/schedulers.py (the easiest way is to just "return []").

You may also want to remove other uninteresting builders to speed up startup.

config/status.py

This is the file that sends notifications of broken builds to all channels (mail, IRC, etc). We don't want that to work, or we'll be spamming a lot of people with our internal stuff.

The easiest way to do this is to modify the getReporters() function to always return [], and remove/comment the "status_email = ..." line.

If you want to play with these settings, so you can receive emails for your own builds, go ahead and try. But make sure no one else gets those emails.

If you don't use custom config file, you should probably comment out everything in this file.

config/auth.py

You probably don't need any authentication:

from buildbot import www

def getAuth():
 return www.auth.NoAuth()

You also might want to remove everything from allowRules in getAuthz method so you can have anonymous access for i.e. stopping builds or force building actions.

Validate installation

The quickest way to check that your config is sane is to run:

$ buildbot checkconfig ~/buildmaster/master.cfg

You may also have to upgrade the database, via:

$ buildbot upgrade-master ~/buildmaster

Since the LLVM config hasn't gone in on the test master database, the property "password" in config/workers.py won't exist, and that will generate an error. You can either set it up (as on the list), or just hardcode it to a fixed password on config/workers.py, so that all your local bots use the same.

Start/Stop the buildmaster

If all is good, you should just start the master and twisted should report no failures. If the last message is something like "server not started" or "server took too long to start", there was an error. 

Errors about not finding the right class can be either that Zorg is not available on the path (symlink or global Python settings), or that the buildbot or some of its components versions is too new. Use pip and debug the error messages.

To start the master, just type:

$ buildbot start ~/buildmaster

If you experience errors, you should also stop it, since it could be left hanging and you won't be able to start it again:

$ buildbot stop ~/buildmaster

If you hit the URL you should see something like this. You should also check twistd.log and keep an eye out for errors that indicate that certain packages, such as subversion, are missing. Otherwise, the buildmaster will appear to be working but will never send any work to the workers.

Updating buildmaster

When you want to pickup changes from upstream buildmaster do roughly this:

buildbot stop ~/buildmaster
git -C ~/llvm-zorg stash
git -C ~/llvm-zorg pull
git -C ~/llvm-zorg stash pop
<resolve conflicts>
git -C ~/llvm-zorg add <conflict files>
buildbot start ~/buildmaster
# If starting fails, check ~/buildmaster/twistd.log for what's wrong.
# "buildbot upgrade-master ~/buildmaster" might solve some migration issues.