Setting up an LLVM Build Worker

Owned by Former user (Deleted)
Last updated: Jul 04, 2022 by David SpickettVersion comment
5 min read

After you have setup a local master, you need to add your own builders and run them locally before submitting it to the public buildbots.

There are three steps to add a new builder, not all of them compulsory.

Adding a new Builder

If you're trying to add something completely new, or changing a builder to add more functionality to it, you'll have to update Zorg. In the repository, there is a directory where all the builders are:

zorg/zorg/buildbot/builders

In there, builders would be on their own Python files and should contain factories that will return a builder by adding all necessary steps to a BuildFactory and returning it. Remember that those steps are built at compile time, so if you need run-time information, you can't inspect the state of the machine, you'll have to use properties. There are plenty of examples in the existing builders.

Once the builder is changed or added, you can use it by replacing the factory parameter when adding the builder to the master. If you added a new builder, you'll have to add an import for it on the builders.py below.

Adding the Builder to the Master

Assuming your builder is correct and accessible, you should add your worker to the master's configuration and connect a builder to it.

Adding a worker

To add a worker, edit the configuration file:

~/buildmaster/config/workers.py

Add your new worker machine by simply adding a new line:

create_worker("linaro-dragon-01", properties={'jobs' : 4}, max_builds=1),

That means 4 threads (-j4) and one build at a time. Since this is a quad-core machine and the build normally uses 100% of the system, that's a good value.

Other possible values is "jobs=2, builds=2", which would underutilise the board if only one build is active at a time, but would be a good value if both builds are always running and they can take a bit longer (for example, on odd configurations that we only need to test once a week or less).

Another idea is to keep jobs=4 and builds=1 but connect two builders on the same bot on the builders.py file (see below). This would intercalate the builds based on each builder's queue. If one queue is empty, it builds again the same builder with the new commits. This is useful for a number of quick builds (without "make clean", or running "ccache"), but it reduces the utility of a fast build.

Adding a builder

Edit the file:

~/buildmaster/config/builders.py

Add a builder for your machine, for example:

{'name': "clang-cmake-thumbv7-a15-full",
 'workernames':["linaro-dragon-01"],
 'builddir':"clang-cmake-thumbv7-a15-full",
 'category': 'sanitizer',
 'factory' : ClangBuilder.getClangCMakeBuildFactory(
                      jobs=4,
                      clean=False,
                      env={'PATH':'/usr/lib/ccache:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'},
                      extra_cmake_args=["-DCMAKE_C_FLAGS='-mcpu=cortex-a15 -mfpu=vfpv3 -mthumb'",
                                        "-DCMAKE_CXX_FLAGS='-mcpu=cortex-a15 -mfpu=vfpv3 -mthumb'",
                                        "-DCOMPILER_RT_TEST_COMPILER_CFLAGS='-mcpu=cortex-a15 -mfpu=vfpv3 -mthumb'",
                                        "-DLLVM_TARGETS_TO_BUILD='ARM;AArch64;X86'"])},

Quick explanation of the parameters:

  • name: is what's going to be referenced on the website, example: http://lab.llvm.org:8011/builders/clang-cmake-armv7-a15
  • workernames: you can have more than one, if they're all identical. We have that for Pandas.
  • builddir: where in ~/buildworker it'll build. This allows you to have different builds on the same machine.
  • category: defines what repositories are monitored for changes. You'll have to look at the official config files to know that, but basically, "sanitizer" means llvm+clang+compiler-rt, while "clang" means only clang+llvm, etc.
  • factory: which builder to use. That above is my own CMake builder. The arguments are:
    • jobs: maximum number of jobs this specific builder will use. The "jobs" property on the builder (above) limits this number, but you can give priorities inside the same machine.
    • clean: runs "make clean" or not
    • env: self explanatory
    • extra_cmake_args: self explanatory, can be found at: http://llvm.org/docs/CMake.html

Adding a new Worker

To add a new worker you have to install buildworker on it. Unlike the master, the versions of buildbot and dependencies on the worker don't matter. Just install it via your package manager.

A quick and simple explanation of how to do it is in the official documentation, just remember to change that to your local builder or the worker will not connect unless the configuration is already reloaded in the public master. Also, don't forget to edit the info/host and info/admin files to match your email and a simple description of the board. This will be public, so give as much information as you can, but nothing private.

Also, the LLVM admins belong to the buildworker group, so they can read/write to the build directories directly without needing sudo (most importantly to list files inside a directory). For that reason, you need to change the umask on the create-worker command line:

# Make sure you run this as the buildworker user!
$ buildbot-worker create-worker --umask=022 ~buildworker/buildworker workerN.llvm.validation.linaro.org <buildworker-name> <password>

Note that the <buildworker-name> and <password> are the one you used in master settings in config/workers.py.

The worker will be compiling LLVM and other modules, so it'll need the minimum requirements like a suitable compiler, libraries and other tools to run the tests. Buildbot already needs python, so the make check should just work (it needs Python 2.7). The other packages you'll need are:

  • build-essential (gcc, g++, binutils - preferably gold, etc)
  • cmake, ninja-build, subversion
  • libxml2-dev, zlib1g-dev, libtinfo-dev

And, if you need to run the test-suite:

  • python-virtualenv
  • bison groff gawk flex tclsh

To install buildworker and the dependencies, use Python's PIP, not apt-get:

$ pip install twisted==12.0 buildbot-worker==0.8.6 'sqlalchemy <= 0.7.10'
 

To start the worker, just type:

$ buildbot-worker start <workerdir>

and you should see it online on your BuildMaster URL. If there are no commits it won't start building, but you can always press the build button on the bot interface to see what happens.

You may have to cycle a few iterations to get all dependencies and environment configurations correct, but once it's going on its own, you should just see it build when it has to. One of the problems that you may run into if your buildworker runs the test-suite is this locale issue with pip. A workaround that we have used in the past is to export LC_ALL=C in the .bashrc of the buildworker user (and also of your own user if you're going to use `bot start`, since it doesn't pick up the right environment at the moment).

Submitting it upstream

Once the bot is working locally, preferably green, and if the bot can be sent upstream (ie. no private hardware/software), it should be sent upstream, so that people can be emailed when they break your bot. That is the main reason why buildbots are useful.

To do that, copy your changes back to the repository, separating patches to the builders and config in different branches (and pull requests), and making sure that config would work on your local build (or even another test build). In case everything works, submit a patch to https://reviews.llvm.org/.

The buildmaster owner is Galina Kristanova, and once she picks up the config on to the master (either by you committing it or her), she'll restart the server and only then your changes will be available to all builders. So, you need to submit the requests in order, first changes to the builders, then changes to the configuration, then restart your buildworker pointing it to the public LLVM master instead of your local installation.