Prepare

Bigtop builds components in containers based on target ${OS}-${ARCH} docker images.

At present Bigtop supports those ${OS}-${ARCH} combinations offcially

OS: debian-9  ubuntu-16.04   centos-7  fedora-26  opensuse-42.3
ARHC: x86_64   ppc64le   aarch64

Before going to build stage, target OS/arch combination docker images, bigtop/puppet and bigtop/slaves should be created. If not, please refer Create build images

Build

Brief

Bigtop uses gradle to build and package components.

Build script for each components can be found at <BIGTOP_SRC_TOP>/bigtop-packages/src/common/<package>/do-component-build.

Debian packaging configurations and scripts are located at <BIGTOP_SRC_TOP>/bigtop-packages/src/deb/<component>.

RPM packaging configurations and scripts are located at <BIGTOP_SRC_TOP>/bigtop-packages/src/rpm/<component>.

There are several taks for a certain component: clean, pkg, deb, rpm.

clean: clean up build
pkg: create tarball package
deb: create deb package
rpm: create rpm package

Here is an example:

./gradlew hadoop-clean                # clean existing hadoop build outputs
./gradlew hadoop-pkg                  # build and create hadoop release tarball
./gradlew hadoop-deb                  # build and create hadoop release deb package. Note: pkg will also be created
./gradlew hadoop-rpm                  # build and create hadoop release rpm package. Note: pkg will also be created

Following cmd can be used to get detail gradle tasks info:

./gradlew tasks

Build components

The recommend build process is to use docker container from target OS-ARCH image to build specified components.

Here is an example to build hbase for ubuntun-16.04/aarch64 platform ( BIGTOP_SRC_TOP is the top of cloned bigtop source dir):

$ cd ${BIGTOP_SRC_TOP}
$ docker images                         # check if required bigtop/slaves:trunk-ubuntun-16.04-aarch64 is present
REPOSITORY          TAG                          IMAGE ID            CREATED             SIZE
bigtop/slaves       trunk-ubuntu-16.04-aarch64   eb812af63515        7 hours ago         3.039 GB
bigtop/puppet       ubuntu-16.04-aarch64         82ef2c36b6e6        7 hours ago         311.5 MB
aarch64/ubuntu      16.04                        a7d1ddc47ced        12 weeks ago        109.7 MB
$ docker run --rm -v `pwd`:/ws bigtop/slaves:trunk-ubuntu-16.04-aarch64 bash -l -c 'cd /ws ; ./gradlew hbase-deb'

Then bigtop will fetch source code tarball from specified URL, applying patches, created patched source tarball/deb/srpm, build and package.

If everything goes well, target package can be found at BIGTOP_SRC_TOP/output/<package>

$ ls output/hbase
hbase_1.1.9-1_all.deb        hbase_1.1.9-1.debian.tar.xz   hbase_1.1.9.orig.tar.gz       hbase-regionserver_1.1.9-1_all.deb
hbase_1.1.9-1_arm64.build    hbase_1.1.9-1.dsc             hbase-doc_1.1.9-1_all.deb     hbase-rest_1.1.9-1_all.deb
hbase_1.1.9-1_arm64.changes  hbase_1.1.9-1_source.changes  hbase-master_1.1.9-1_all.deb  hbase-thrift_1.1.9-1_all.deb